Systems and methods for coaching a user for game play

ABSTRACT

A method for processing a self-coaching interface is described. The method includes identifying a gameplay event during gameplay by a user. The gameplay event is tagged as falling below a skill threshold. The method further includes generating a recording for a window of time for the gameplay event and processing game telemetry for the recording of the gameplay event. The game telemetry is used to identify a progression of interactive actions before the gameplay event for the window of time. The method includes generating overlay content in the self-coaching interface. The overlay content is applied to one or more image frames of the recording when viewed via the self-coaching interface. The overlay content appears in the one or more image frames during a playback of the recording. The overlay content provides hints for increasing a skill of the user to be above the skill threshold.

FIELD

The present disclosure relates to systems and methods for coaching auser for game play.

BACKGROUND

It is common that video games are designed to present challenges forplayers to figure out how to overcome. In some games, this may be achallenge of besting another player in a multiplayer setting, and inother games, this may be a challenge of defeating a difficultcomputer-controlled entity. Typically, the players attempt thesechallenges multiple times, and slowly learn several intricacies of thegames to improve their skill. Additionally, some of the most challengingsituations may not occur often, making it harder for the players toremember lessons learned during previous encounters. Some players maybecome frustrated with the amount of time for improving their skill at aparticular game.

It is in this context that embodiments of the invention arise.

SUMMARY

Embodiments of the present disclosure provide systems and methods forcoaching a user for game play.

In an embodiment, a coaching system is provided to reduce frustration ofplayers of various game genres in learning a new game. The coachingaccelerates skill improvement for the players.

In one embodiment, a series of gameplay events, such as death of avirtual user and an attack missed by the virtual user, are recorded overtime and collected into grouped sequences. The systems and methods,described herein, provide a user interface (UI) to review video andcontroller input capture of previous failures in specific situations toexamine and learn what alternative strategy should have been used. Thesystems and methods highlight the failures that occur during a play ofthe game.

In an embodiment, the systems and methods described herein use gamehooks or telemetry to identify points, such as gameplay events, duringthe game where coaching can be applied. For example, coaching can beapplied after death of the virtual user or after an attack on thevirtual user. A recording of the game for a last X amount of time unitsbefore a gameplay event is saved in a memory device, where X is apositive real number. The recording is saved with a tag describing anissue, such as death of the virtual user or attack on the virtual user.The systems and methods generate a dedicated UI to replay the recordingwith overlays, such as input selections received from the user. Also, atimeline scrubber is provided along the dedicated UI to allow the userto scroll backward and forward during the last X amount of time units.Also, the dedicated UI includes hints on mechanics or situations toprovide to the user to increase a skill level of the user.

In one embodiment, a method for processing a self-coaching interface isdescribed. The method includes identifying a gameplay event duringgameplay by a user. The gameplay event is tagged as falling below askill threshold. The method further includes generating a recording fora window of time for the gameplay event and processing game telemetryfor the recording of the gameplay event. The game telemetry is used toidentify a progression of interactive actions before the gameplay eventfor the window of time. The method includes generating overlay contentin the self-coaching interface. The overlay content is applied to one ormore image frames of the recording when viewed via the self-coachinginterface. The overlay content appears in the one or more image framesduring a playback of the recording. The overlay content provides hintsfor increasing a skill of the user to be above the skill threshold.

In an embodiment, a method facilitating self-coaching of a player duringa play of a game is described. The method includes recording a series ofgameplay events as game event data. The series of gameplay events isassociated with a user identity (ID). The method further includesclassifying the series of gameplay events based on a plurality ofpredetermined gameplay events and receiving one or more inputs toinitiate a coaching session regarding one of the plurality ofpredetermined gameplay events. In response to the one or more inputs,the method includes obtaining a portion of the game event data. Theportion of game event data is recorded within a predetermined timeperiod of occurrence of the one of gameplay events of the series. Theone of the gameplay events of the series is classified as one of theplurality of predetermined events. The method includes displaying theportion of the game event data with a timeline bar to facilitate theself-coaching of the player.

In an embodiment, the series of gameplay events are associated with theuser identity when the game having the series of gameplay events isaccessed after authenticating the user identity.

In one embodiment, the method also includes receiving an inputindicating a time along the timeline bar and translating the portion ofgame event data according to the input indicating the time.

In an embodiment, the method also includes recording an additionalseries of gameplay events as additional game event data. The additionalseries of gameplay events is associated with the user ID. The methodincludes classifying the additional series of gameplay events based onthe plurality of predetermined gameplay events. In response to the oneor more inputs, the method includes obtaining a portion of theadditional game event data. The additional portion of game event data isrecorded within the predetermined time period of occurrence of the oneof gameplay events of the additional series. The one of the gameplayevents of the additional series is classified as one of the plurality ofpredetermined events. The method includes displaying the portion of theadditional game event data with the timeline bar to facilitate theself-coaching of the player.

In one embodiment, the method includes recording an additional series ofgameplay events as additional game event data. The additional series ofgameplay events is associated with another user ID. The method includesclassifying the additional series of gameplay events based on theplurality of predetermined gameplay events. In response to the one ormore inputs, the method includes obtaining a portion of the additionalgame event data. The additional portion of game event data is recordedwithin the predetermined time period of occurrence of the one ofgameplay events of the additional series. The one of the gameplay eventsof the additional series is classified as one of the plurality ofpredetermined events. The method includes displaying the portion of theadditional game event data with the timeline bar to facilitate theself-coaching of the player.

Some advantages of the herein described systems and methods includeproviding a user interface that enables users to understand a video gameand quickly achieve skills for playing the game. Also, the video game isimproved because a coaching session is associated with the video gameand accessed to allow the user to self-coach. Furthermore, unnecessarynetwork traffic is reduced. When the user applies lessons learned duringthe coaching session to play the video game, the user can play the videogame by making less errors compared to those made without applying thecoaching session. These errors unnecessary increase the network trafficbetween a client device used by the user and a server system in whichthe video game is executed.

Other aspects of the present disclosure will become apparent from thefollowing detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principles ofembodiments described in the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present disclosure are best understood byreference to the following description taken in conjunction with theaccompanying drawings in which:

FIG. 1 is a diagram of an embodiment of a system to illustrate a play ofa game.

FIG. 2 is a diagram of an embodiment of the system to illustrate avirtual scene of the game.

FIG. 3 is a diagram of an embodiment of the system of FIG. 1 toillustrate another virtual scene of the game.

FIG. 4 is a diagram of an embodiment of the system of FIG. 1 toillustrate yet another virtual scene in which a virtual user dies.

FIG. 5A is a diagram of an embodiment of a system to illustraterecording of game event data by a game recorder for a user ID1.

FIG. 5B is an embodiment of the game recorder to illustrate recording ofgame event data for user IDs 2 through N.

FIG. 6 is a diagram of an embodiment of a system to illustrate use of acoaching program to generate a coaching session for training a user.

FIG. 7A is a diagram of an embodiment of a coaching scene forillustrating overlays of one or more virtual objects on one or morevirtual objects of a virtual scene of FIG. 1.

FIG. 7B is a diagram of an embodiment of a coaching scene that isrendered and displayed on a display device of the system of FIG. 1.

FIG. 8A is a diagram of an embodiment to illustrate a simultaneousdisplay of the coaching scene of FIG. 7A and another coaching scene onthe display device of FIG. 1.

FIG. 8B is a diagram of an embodiment of the display device of FIG. 1 toillustrate a simultaneous rendering and display of the coaching scene ofFIG. 7 associated with the user ID1 and another coaching sceneassociated with the user ID N.

FIG. 8C is a diagram of an embodiment of a system to illustrate thatinstead of or in addition to rendering a coaching scene on a displayscreen, a processor generates audio data that is output as sound to auser via a head phone.

FIG. 9 is a diagram of an embodiment of a system to illustrate use of aninferred training engine for generating the coaching session of FIG. 6.

FIG. 10 is a flowchart to illustrate an embodiment of a method fortraining a model.

FIG. 11A is a diagram of an embodiment of a system to illustrate acommunication via a router and modem and a computer network between aprocessor system and multiple devices.

FIG. 11B is a diagram of an embodiment of a system to illustrate thatthe processor system can be implemented within a game console.

FIG. 11C is a diagram of an embodiment of a system to illustratecommunication between a smart phone and the processor system via thecomputer network without using the router and modem between the computernetwork and the smart phone.

FIG. 12A is a diagram of an embodiment of a headphone.

FIG. 12B is a diagram of an embodiment of a display device fordisplaying an image of a coaching scene or a virtual scene on a displayscreen.

FIG. 13 is a flow diagram conceptually illustrating various operationswhich are performed for streaming a cloud video game to a client device,in accordance with various implementations of the present disclosure.

FIG. 14 is a block diagram of an embodiment of the game console that iscompatible for interfacing with a display device of the client deviceand is capable of communicating via the computer network with a gamehosting system.

FIG. 15 is a diagram of an embodiment of a head-mounted display (HMD) toillustrate components of the HMD.

FIG. 16 illustrates an embodiment of an Information Service Provider(INSP) architecture.

DETAILED DESCRIPTION

Systems and methods for coaching a user for game play are described. Itshould be noted that various embodiments of the present disclosure arepracticed without some or all of these specific details. In otherinstances, well known process operations have not been described indetail in order not to unnecessarily obscure various embodiments of thepresent disclosure.

FIG. 1 is a diagram of an embodiment of a system 100 to illustrate aplay of a game. The system 100 includes a display device 1 that displaysa virtual scene 110 of the game. The display device 1 has a camera 101that captures one or more images of a real-world environment in front ofthe camera 101. For example, the camera 101 captures one or more imageswithin a field-of-view of the camera 101. As an example, each virtualscene is an image frame. In one embodiment, the terms image and imageframe are used herein interchangeably. Examples of a display device, asused herein, include a smart television, a television, a plasma displaydevice, a liquid crystal display (LCD) device, and a light emittingdiode (LED) display device. A user 1 is playing the game, such as avideo game, by using a game controller 1. Examples of the video gameinclude a single player game or a multi-player game. Examples of a gamecontroller 1 include a Playstation™ controller, an Xbox™ controller, anda Nintendo™ controller. As an example, a game controller includesmultiple joysticks and multiple buttons.

The display device 1 displays multiple images of the game. The user 1logs into a user account 1 that is assigned to the user 1 by a serversystem to access the game from the server system. For example, a useridentity (ID) assigned to a user and a password assigned to the user bythe server system are authenticated by the server system to determine toallow the user to access the game.

Upon accessing the game, the user 1 plays the game by using the gamecontroller 1. During the play of the game, the virtual scene 110 isgenerated and a user ID 1 assigned to the user 1 is displayed in thevirtual scene 110.

In the game, the user 1 uses the game controller 1 to build a virtualramp 112 on which a virtual user 102 can climb to have a heightadvantage over another virtual user 104. The virtual user 102, thevirtual ramp 112, and the virtual user 104 are within the virtual scene110. Also, in the game, movement of the virtual user 102 is controlledby the user 1 with the game controller 1. The user 1 uses the gamecontroller 1 to control a virtual gun 108 that is held by the virtualuser 102. For example, the user 1 uses the game controller 1 to controlthe virtual gun 108 to shoot at the virtual user 104. The virtual gun108 is a portion of the virtual scene 110. The virtual user 104 iscontrolled by another user, such as a user 2 or a user 3, in themulti-player game.

During the play of the game, in the virtual scene 110, a virtual bullet106 is directed towards the virtual user 102 while the virtual user 102is shooting at the virtual user 104 and the virtual user 104 is shootingat the virtual user 102. The virtual bullet 106 is directed towards tothe virtual user 102 from a side of the virtual user 102. The virtualscene 110 does not include a virtual user that shot the virtual bullet106 at the virtual user 102. The virtual scene 110 includes a virtualtree 105 and a virtual wall 107. It should be noted that each of thevirtual tree 105, the virtual wall 107, the virtual ramp 112, thevirtual user 102, the virtual gun 108, the virtual bullet 108, and thevirtual user 104 is an example of a virtual object.

In one embodiment, the server system authenticates a user ID and apassword to allow a user to access multiple games for game play.

In an embodiment, instead of the display device 1, a head-mounteddisplay (HMD) is used. The HMD is worn by the user 1 on his/her head.

FIG. 2 is a diagram of an embodiment of the system 100 to illustrate avirtual scene 202 of the game. The virtual scene 202 is displayed on thedisplay device 1 and follows the virtual scene 110 during the play ofthe game. For example, both the virtual scenes 110 and 202 are displayedduring the same gaming session. The user 1 is not able to save thevirtual user 102 from being hit by the virtual bullet 106 (FIG. 1).After the virtual bullet 108 hits the virtual user 102, the virtual user102 dies in the game and is beamed by a virtual drone 204 of the virtualscene 202. Also, before the virtual user 102 dies in the game, the user1 manages to control the game controller 1 to further control thevirtual user 102 and the virtual gun 108 (FIG. 1) to kill the virtualuser 104. When the virtual user 104 is killed, the virtual user 104 isbeamed by another virtual drone 206 in the virtual scene 202. Also, inthe virtual scene 202, the user ID1 is displayed.

FIG. 3 is a diagram of an embodiment of the system 100 to illustrateanother virtual scene 302 of the game. The virtual scene 302 isdisplayed after the virtual scene 202 (FIG. 2) is displayed or beforethe virtual scene 110 (FIG. 1) is displayed. In the virtual scene 302,the virtual user 102 is controlled by the user 1 via the game controller1. The virtual scene 302 is displayed on the display device 1. Thevirtual user 102 is controlled to use the virtual gun 108 to shoot atthe virtual user 104. Also, the virtual scene 302 includes a virtualwall 304 built by the virtual user 102. For example, the user 1 uses thegame controller 1 to control the virtual user 102 to build the virtualwall 304 to protect the virtual user 102 from virtual bullets being shotby the virtual user 104.

In the virtual scene 302, a virtual bullet 306 hits the virtual user 102from behind. The user 1 cannot see, from the virtual scene 302, whichvirtual user shot the virtual bullet 306 at the virtual user 102. Thevirtual scene 302 does not include a virtual user that shot the virtualbullet 306 at the virtual user 102.

It should be noted that the virtual scene 302 is displayed on thedisplay device 1 during a gaming session that is the same or differentfrom a gaming session in which the virtual scenes 110 and 202 of FIGS. 1and 2 are displayed. For example, after the virtual user 102 dies in thevirtual scene 202, a game program of the game is executed by a processorof the server system to provide a waiting time period in which the user1 waits for the virtual user 102 to respawn or rejuvenate or to comealive. An example of a processor, as used herein, includes amicroprocessor or a microcontroller or a central processing unit (CPU)or an application specific integrated circuit (ASIC) or a programmablelogic device (PLD). The waiting time period occurs before the gamedisplays the virtual scene 302. In this example, the user 1 does not logout of the user account 1 between the display of the virtual scenes 202and 302, and so the virtual scenes 110, 202, and 302 are displayedduring the same gaming session. As another example, after the virtualuser 102 dies in the virtual scene 202, the user 1 logs out of the useraccount 1. The user 1 later logs into the user account 1. After the user1 logs into the user account 1, the virtual scene 302 is displayed. Inthis example, the virtual scene 302 is displayed during a gaming sessionthat is different from the gaming session in which the virtual scenes110 and 202 are displayed.

In one embodiment, a gaming session ends at a time the virtual user 102that is controlled by the user 1 either dies or wins the gaming session.The virtual user 102 wins the gaming session when the virtual user 102kills all other virtual users in the gaming session or survives beyond aprestored time period during the gaming session. Once the virtual user102 dies, the virtual user 102 can be respawned and another gamingsession begins, and the other gaming session ends either when thevirtual 102 dies or wins the other gaming session.

FIG. 4 is a diagram of an embodiment of the system 100 to illustrate yetanother virtual scene 402 in which the virtual user 102 dies. Thevirtual scene 402 is displayed after the virtual scene 302 (FIG. 3) isdisplayed. For example, both the virtual scenes 302 and 402 aredisplayed during the same gaming session. In the virtual scene 402, thevirtual users 102 and 104 are killed. The virtual user 102 is killed bythe virtual bullet 306 (FIG. 3). The virtual scene 402 includes thevirtual drones 204 and 206 used to beam in the virtual users 102 and104.

FIG. 5A is a diagram of an embodiment of a system 500 to illustrate therecording of game event data 510 by a game recorder 502. The system 500includes a processor system 505. An example of a processor system 505 isa server system, which includes one or more servers of a data center orof multiple data centers. Another example of the processor system 505includes one or more processors of the server system. Yet anotherexample of the processor system 505 includes one or more virtualmachines. An example of the game recorder 502 is a digital videorecorder that records both video and audio data of the game event data510. The processor system 505 includes the game recorder 502, a gameprocessor 506 and a coaching processor 508. The game processor 506 iscoupled to the game recorder 502 and to the coaching processor 508. Thecoaching processor 508 is coupled to the game recorder 502.

The game processor 506 executes a game program 528 to facilitate theplay of the game by the user 1 and by other users. As an example, thegame program 528 includes a game engine and a rendering engine. The gameengine is executed for determining positions and orientations of variousvirtual objects of a virtual scene. The virtual scene includes abackground, which is an example of the virtual object. The renderingengine is executed for determining graphics parameters, such as color,intensity, shading, and lighting of the virtual objects of the virtualscene. The game program 528 is stored in one or more memory devices ofthe processor system 502 and the one or more memory devices are coupledto the game processor 506. When the game program 528 is executed, thegame event data 510 is generated and the virtual scenes 110, 220, 302,and 402 (FIGS. 1-4) are displayed on the display device 1.

The game event data 510 generated is for the user ID1. For example, thevirtual scenes 110 (FIG. 1), 202 (FIG. 2), 302 (FIG. 3), and 402 (FIG.4) of the game event data 510 are generated upon execution of the gameprogram 528. The game program 528 is executed when the user ID 1 andother information, such as a password or a phone number or a combinationthereof, of the user 1 are authenticated by the processor system 505.

As an example, the game event data 510 includes multiple game events 1through m leading up to a death 1 of the virtual user 102 is illustratedin the virtual scene 202 (FIG. 2), where m is a positive integer. Toillustrate, the game event data 510 includes data of the virtual scenes110 (FIG. 1) and 202. To further illustrate, a game event 512 is data ofthe virtual scene 110 and a game event 514 is data of the virtual scene202. As another example, the game event data 510 includes multiple gameevents 1 through n leading up to a death x of the virtual user 102illustrated in the virtual scene 402 (FIG. 4), where n and x arepositive integers. To illustrate, the game event data 510 includes dataof the virtual scenes 302 (FIG. 3) and 402. To further illustrate, agame event 516 is data of the virtual scene 302 and a game event 518 isdata of the virtual scene 402. As another example, the game event data510 includes multiple game events 1 through o leading up to a decreasein health level of the virtual user 102 to be below a predeterminedthreshold, where o is a positive integer. To illustrate, a game event520 is data of a virtual scene in which the virtual user 102 has fullhealth and a game event 522 is data of a virtual scene in which thevirtual user 102's health decreases to be below the predeterminedthreshold. As yet another example, the game event data 510 includesmultiple game events 1 through p leading up to a decrease in healthlevel of the virtual user to be below the predetermined threshold, wherep is a positive integer. To illustrate, a game event 524 is data of avirtual scene in which the virtual user 102 has full health and a gameevent 526 is data of a virtual scene in which the virtual user 102'shealth decreases to be below the predetermined threshold. It should benoted that a series of game events from the game event 520 to the gameevent 522 occurs during the same or a different gaming session then anoccurrence of a series of game events from the game event 524 to thegame event 526.

During execution of the game program 528, the game recorder 502 recordsthe game event data 510. For example, a processor of the game recorder502 stores or writes the game event data 510 to one or more memorydevices of the game recorder 502. Examples of the memory device includea read-only memory, a random access memory, and a combination thereof.To illustrate, the memory device is a flash memory device or a redundantarray of independent disks (RAID).

During or after execution of the game program 528, the game event data510 that is recorded is provided from the game recorder 502 to thecoaching processor 508. For example, the coaching processor 508periodically requests the game recorder 502 for obtaining the game eventdata 510 from the game recorder 502. In response to reception of arequest from the coaching processor 508, the game recorder 502 accesses,such as reads, the game event data 510 from the one or more memorydevices of the game recorder 502 and sends the game event data 510 tothe coaching processor 508. As another example, the game recorder 502periodically sends the game event data to the coaching processor 508. Toillustrate, without receiving any request from the coaching processor508, the game recorder 502 accesses the game event data 510 from the oneor more memory devices of the game recorder 502 and sends the game eventdata 510 to the coaching processor 508.

During or after execution of the game program 528, a coaching program530 is executed by the coaching processor 508. For example, the coachingprogram 530 is executed after the display of the virtual scene 202 (FIG.2) and before the display of the virtual scene 302 (FIG. 3) or after thedisplay of the virtual scene 402 (FIG. 4).

It should be noted that the coaching program 530 performs the functionsdescribed herein as being performed by the coaching program 530 when thecoaching program 530 is executed by the processor system 505. It shouldfurther be noted that in an embodiment instead of the game recorder 502,multiple game recorders are used. Similarly in one embodiment is that ofthe game processor 506, multiple game processors are used. In oneembodiment instead of the coaching processor 508, multiple coachingprocessors are used.

In one embodiment, the game recorder 502, the game processor 506 and thecoaching processor 508 are coupled with each other via a bus.

In an embodiment, the coaching program 530 is a portion of the gameprogram 528 and is integrated into the game program 528. For example, acomputer program code of the coaching program 520 is interspersed with acomputer program code of the game program 528.

In one embodiment, a game event is sometimes referred to herein as agameplay event.

In an embodiment, each time a death of the virtual user 102 occurs, thecoaching program 530 identifies from the game event data 510, that thedeath has occurred. For example, the coaching program 530 determinesthat the virtual scene 202 includes the virtual drone 204 above thevirtual user 204 to determine that the virtual user 102 has died. Thecoaching program 520 tags, such as assigns a keyword, such as death ordemise, to the virtual scene 202 in which the virtual user 102 dies, andstores the keyword as metadata within one or more memory devices of theprocessor system 505. Each time the virtual user 102 dies, the coachingprogram 530 determines that a skill level of the user 1 is below apreset threshold, which is stored in one or more memory devices of theprocessor system 505. For example, each time the virtual user 102 diesin the game, the game program 528 reduces a skill level corresponding tothe user ID1 to be below the preset threshold.

FIG. 5B is an embodiment of the game recorder 502 to illustraterecording of game event data 504 for user IDs 2 through N, where N is apositive integer. The user IDs 2 through N are assigned to other users 2through N that are playing the game. For example, the user ID2 isassigned to a user 2 and the user IDN is assigned to a user N.

The game event data 504 includes a series of game events 1 through q forthe user IDX and another series of game events 1 through r recorded forthe user IDN. For example, a game event 550 includes data for a virtualscene that is accessed upon execution of the game program 528 (FIG. 5A).The game event q includes a death q of the virtual user 102. The gameprogram 528 is accessed after the user IDN is authenticated by theprocessor system 505 (FIG. 5A). As another example, a game event 552includes data for a virtual scene that is accessed upon execution of thegame program 528. The game event 552 includes a death 1 of a virtualuser that is controlled by the user N via a game controller N. Asanother example, a game event 554 includes data for a virtual scene thatis accessed upon execution of the game program 528 and a game event 556includes data for a virtual scene that is accessed upon execution of thegame program 528. The game event 556 includes a death r of the virtualuser that is controlled by the user N via the game controller N, where ris a positive integer. During or after execution of the game program528, the game event data 504 that is recorded is provided from the gamerecorder 502 to the coaching processor 508 (FIG. 5A) in the same mannerin which the game event data 510 (FIG. 5A) is provided from the gamerecorder 502 to the coaching processor 508.

FIG. 6 is a diagram of an embodiment of a system 600 to illustrate a useof the coaching program 530 to generate a coaching session 608 fortraining the user 1. The system 600 includes the game event data 510 forthe user ID1. Also, the system 600 includes the coaching program 530,the game event data 504 for the user IDs 2-N, a client device 602,selected game event data 604, and metadata 606. Examples of a clientdevice, as used herein, include a desktop computer, a laptop computer, atablet, a smart phone, a combination of a hand-held remote controllerand smart television, a combination of a hand-held controller (HHC) anda game console, and a combination of a hand-held controller and a headmounted display (HMD). A game controller is an example of the HHC. Thedisplay device 1 is an example of a display device of the client device602.

The game event data 510 for the user ID1 includes video data 612 andaudio data 614. As an example, video data of game event data includes aposition, an orientation, and the graphics parameters of multiplevirtual objects displayed in a virtual scene. Also, as an example, audiodata of game event data includes phonemes, vocabulary, lyrics, orphrases that are uttered by one or more virtual objects in a virtualscene. Similarly, the game event data 504 for the user IDs 2-N includesvideo data and audio data for each of the user IDs 2-N.

Examples of the metadata 606 include game telemetry, such as one or moregame states for generating virtual objects in the game. For example, themetadata 606 includes a game state based on which a virtual object,which is not displayed or output or uttered in a virtual scene, isdisplayed in a coaching scene. As another example, the metadata 606 isinformation associated with positions and orientations of one or morevirtual objects in a virtual scene, such as the virtual scene 110 or 202or 302 or 402 (FIGS. 1-4), that is displayed during execution of thegame program 528. To illustrate, the metadata 606 is a position at whicha virtual rectangular frame is displayed around a position of thevirtual bullet 106 that is displayed in the virtual scene 110. Thevirtual rectangular frame is not displayed in the virtual scene 110. Asanother example, the metadata 606 is a position and orientation of avirtual user that is not displayed in the virtual scene 110 or 202 or302 or 402 (FIGS. 1-4). The virtual user, not displayed, shoots thevirtual bullet 106 (FIG. 1) or 306 (FIG. 3). The coaching program 530generates the virtual user that shot the virtual user 102 at a positionnext to, e.g., to the right of, a position of the virtual bullet 106 toshow the virtual user as shooting the virtual bullet 106 towards thevirtual user 102. As yet another example, the metadata 606 is a usernameof the virtual user that is not displayed in the virtual scene 110 or202 or 302 or 402 and that shoots a virtual bullet. As another example,the metadata 606 is not used to generate a camera view from view pointof the virtual user that is not displayed in the virtual scene 110 or202 or 302 or 402. For example, the metadata 606 is not used to generatean action replay of the virtual scene 110 or 202 or 302 or 402 from theviewpoint of the virtual user that is not displayed in the virtual scene110 or 202 or 302 or 402.

Examples of the coaching session 608 includes a display of a coachingscene in which the user 1 cannot play the game but can learn based onhis/her previous play of the game, or previous play of the game by theusers 2-N, or a combination thereof. For example, the coaching program530 is executed by the processor system 505 (FIG. 5A) to generate thecoaching session 608 for the user ID1. An example of the coaching sceneis a virtual scene that is generated and rendered upon execution of thecoaching program 530. When the coaching session 608 is generated, one ormore coaching scenes are displayed on the client device 602. Thecoaching scenes, when displayed for a user ID, do not allow a play ofthe game to the user ID. For example, during a display of the coachingscenes for the user ID1, the game program 528 (FIG. 5A) is not executedby the processor system 505 for the user ID1. To illustrate, for theuser ID1, the processor system 505 does not render a virtual scene thatis generated by executing the game program 528 during a display of oneor more coaching scenes for the user ID1. Instead, the processor system505 executes the coaching program 530 for rendering the one or morecoaching scenes for the user ID1.

In an operation 610, the coaching program 530 receives an input signal616 from the client device 602 to initiate the coaching session 608. Forexample, during a play of the game, the user 1 selects a button on theclient device 1 to generate the input signal 616 for initiating thecoaching session 608. To illustrate, after the virtual user 102 dies inthe virtual scene 202 or 402 (FIG. 2 or 4), the user 1 selects a buttonof the game controller 1 (FIG. 1). Upon receiving the selection, thegame controller 1 generates the input signal 616 for initiating thecoaching session 608. The input signal 616 is sent from the clientdevice 602 to the coaching program 530.

The coaching program 530 receives the input signal 616 and identifiesfrom the input signal 616 that the coaching session 608 is to beinitiated. Upon identifying to the coaching session 608 is to beinitiated, the coaching program 530 generates an output signal 618including a request 620 for one or more user IDs 622 for which thecoaching program 520 is to be initiated. For example, the output signal618 includes a query to the client device 602 for obtaining informationregarding whether the coaching program 530 is to be initiated based ongameplay by the user ID1 or user ID2 or user IDN or a combination of twoor more of the user IDs ID1, ID2, and IDN. To illustrate, the outputsignal 618 includes an inquiry for obtaining the user ID1 or user ID2 orthe user IDN or a combination of two or more thereof that is accessedfor gameplay of the game.

The client device 602 receives the request 620 for the one or more userIDs 622 from the coaching program 530. For example, the client device602 displays the request 602 for the one or more user IDs 622 on adisplay screen of the client device 602. To illustrate, the clientdevice 602 displays a list of the user IDs 1-N on the display screen ofthe client device 602. As another example, the client device 602 outputsa sound via a headphone worn by the user 1 asking the user 1 for the oneor more user IDs 622. The headphone is a part of the client device 602or is coupled to the client device 602. As yet another example, theclient device 602 outputs a sound via one or more speakers of the clientdevice 602 asking the user 1 for the one or more user IDs 622.

The client device 602 receives the one or more user IDs 622 from theuser 1 and generates an input signal 623 that includes the one or moreuser IDs 622. For example, the user 1 selects one or more buttons on thegame controller 1 to provide the one or more user IDs 622. Toillustrate, the user 1 selects one or more buttons on the gamecontroller 1 to select one or more of the user IDs 1-N displayed on thedisplay screen of the client device 602. The client device 602 sends theinput signal 623 to the coaching program 530.

Upon receiving the one or more user IDs 622 within the input signal 623,the coaching program 530 generates an output signal 624 including arequest 626 for identifying a game sequence type for which the coachingsession 608 is to be generated for the one or more user IDs 622. Forexample, the request 626 is an inquiry regarding whether the gamesequence type is of a type 1 or a type 2. An example of type 1 of a gamesequence includes multiple game events that lead to a death of a virtualuser and an example of type 2 of a game sequence includes multiple gameevents that lead to a decrease in health level of the virtual user. Toillustrate, the type 1 includes the game events 512 through 514 and thegame events 516 and 518, and the type 2 includes the game events 520through 522 and 524 through 526 (FIG. 5A).

The client device 602 receives the output signal 624 including therequest 626 for identifying the game sequence type and outputs therequest to the user 1 via a user interface or another type of interfaceof the client device 602. For example, the client device 602 displays orrenders the request 626 via a display screen of the client device 602.To illustrate, the client device 602 displays a list of game sequencetypes including a death of the virtual user 102 (FIGS. 1-4) and areduction in a health level of the virtual user 102 below thepredetermined threshold. As another example, the client device 602outputs a sound via one or more speakers of the client device 602 andthe sound includes the request 626 for the game sequence type. As yetanother example, the client device 602 outputs a sound via the headphoneto the user 1 for obtaining a response to the request 626.

In response to the request 626, the user 1 provides a game sequence typeidentifier 628 via a user interface or another type of interface of theclient device 602. For example, the user 1 selects one or more buttonsof the game controller 1 to provide the game sequence type identifier628. To illustrate, the user 1 selects one or more buttons to selecteither the game sequence type of death of the virtual user 102 or thereduction in the health level of the virtual user 102. Upon receivingthe selection of the game sequence type identifier 628, the clientdevice 602 generates an input signal 630 including the game sequencetype identifier 628 and sends the input signal 630 to the coachingprogram 530.

The coaching program 530 receives the input signal 630 and identifiesthe game sequence type identifier 628 from the input signal 630. Uponidentifying the game sequence identifier 628, the coaching program 530generates an output signal 634 including a request 632 for identifying agame event of the sequence type identified by the game sequenceidentifier 628. For example, to determine whether the coaching session608 is to be generated for the game event 514 or 518 (FIG. 5A) of deathof the virtual user 102, the coaching program 530 generates the request632 for identifying the game event 514 or 518. The coaching program 530sends the output signal 634 to the client device 602.

Upon receiving the output signal 634, the client device 602 identifiesthe request 632 from the output signal 634 and provides the request 632via a user interface or another interface of the client device 602 tothe user 1. For example, the client device 602 displays a list of thegame events 514 and 518 on the display screen of the client device 602.To illustrate, the client device 602 displays the virtual scene 202 andthe virtual scene 402 on the display screen of the client device 602 toallow the user 1 to select one of the virtual scenes 202 and 402.

The user 1 responds to the request 632 to identify the game event forwhich the coaching session 608 is to be generated. The game event isidentified to provide a game event identifier 636. For example, the user1 selects via one or more buttons of the game controller 1, the virtualscene 202 to identify the game event 514 or the virtual scene 402 toidentify the game event 518. In response to receiving the selection ofthe virtual scene 202 or 402, the client device 602 generates an inputsignal 638 having the game event identifier 636, and sends the inputsignal 638 to the coaching program 530.

Upon receiving the input signal 638, the coaching program 530 identifiesthe game event identifier 636 and selects the game event data 604 forwhich the coaching session 608 is to be generated for the game user ID622. For example, the coaching program 530 identifies the game eventidentifier 636 identifying the game event 514 (FIG. 5A) recorded basedon the virtual scene 202 (FIG. 2) that is displayed when the gameprogram 528 is executed for the user ID1. An example of the selectedgame event data 604 includes the game event 514 or the game event 518(FIG. 5A).

Upon identifying the game event identifier 636 for which the coachingsession 608 is to be generated, the coaching program 530 is executed bythe processor system 505 to store a portion of the game event data 510that is recorded within a predetermined amount of time interval from atime of recording of the game event identified by the game eventidentifier 636 to output the selected game event data 604. For example,upon identifying the game event 514 for which the coaching session 608is to be generated, the coaching program 530 stores one or more gameevents that are recorded within a predetermined time interval before atime at which the game event 514 is recorded. The coaching program 530also stores the game event 514. The one or more game events lead up toor result in an occurrence of the game event 514. To illustrate, thecoaching program 530 deletes from one or more memory devices of theprocessor system 505 one or more game events that are recorded outsidethe predetermined time interval before the time at which the game event514 is recorded. The game event 514 and the game events within thepredetermined time interval are stored by the coaching program 530 asthe selected game event data 604. Also, in this illustration, theprocessor system 505 retains within the one or more memory devices ofthe processor system 505 one or more game events that are recordedwithin the predetermined time interval before the time at which the gameevent 514 is recorded to retain the selected game event data 604. Theone or more game events that are deleted lead to the one or more gameevents that are retained, and the one or more game events that areretained lead to the game event 514. The one or more game events thatare retained are the selected game event data 604.

To explain further, during a gaming session of the game, a series ofvirtual scenes are generated and recorded upon execution of the gameprogram 528. The series includes a first series, a second series, and athird series. The first series includes a first set of consecutivevirtual scenes in which the virtual user 102 collects items, such asvirtual weapons and virtual objects, such as bricks or metal bars, todefend the virtual user 102 from virtual bullets. The second seriesincludes a second set of consecutive virtual scenes in which the virtualuser 102 fights with other virtual users. The third series includes athird set of consecutive virtual scenes, such as the virtual scenes 110and 202 (FIGS. 1 and 2), for which the game events 512 and 514 (FIG. 5A)are recorded. The second series is consecutive to the first series andthe third series is consecutive to the second series. The processorsystem 505 deletes the first and second series of virtual scenes andretains the third series of virtual scenes. The third series is theselected game event data 604.

As another illustration, the coaching program 530 stores the selectedgame event data 604, which includes a portion of the game event data510, within one or more memory devices of the processor system 505. Theone or more memory devices of the processor system 505 in which theselected game event data 604 is stored are different from, e.g., not thesame as, one or more memory devices in which the game event data 510 isrecorded. In one embodiment, instead of being stored at different one ormore memory device, the selected game event data 604 is stored in amemory device at different memory addresses than memory addresses atwhich the game event data 510 is recorded in the memory device.

As yet another example, upon identifying the game event 514 for whichthe coaching session 608 is to be generated, the coaching program 530stores one or more game events that are recorded within a predeterminedtime interval before a time at which the game event 514 is recorded, oneor more game events recorded within a preset time interval after thetime at which the game event 514 is recorded, and the game event 514.The one or more game events that are recorded within the predeterminedtime interval before the time at which the game event 514 is recorded,the game event 514, and the one or more game events recorded within thepreset time interval after the time at which the game event 514 isrecorded are the selected game event data 604. For example, uponidentifying the game event 514 for which the coaching session 608 is tobe generated, the coaching program 530 stores one or more game eventsthat are recorded within the preset time interval after the time atwhich the game event 514 is recorded. The game event 514 is followed bythe one or more game events recorded within the preset time intervalafter the time at which the game event 514 is recorded. To illustrate,the coaching program 530 deletes from one or more memory devices of theprocessor system 505 one or more game events that are recorded outsidethe preset time interval after the time at which the game event 514 isrecorded. The one or more game events that occur within the preset timeinterval after the time at which the game event 514 is recorded lead tothe one or more game events that are deleted. Also, in thisillustration, the processor system 505 retains within the one or morememory devices of the processor system 505 one or more game events thatare recorded within the preset time interval after the time at which thegame event 514 is recorded.

To explain further, during a gaming session of the game, a series ofvirtual scenes are generated upon execution of the game program 528. Theseries includes a first series, a second series, and a third series. Thefirst series includes a first set of consecutive virtual scenes in whichthe virtual user 102 collects items, such as virtual weapons and virtualobjects, such as bricks or metal bars, to defend the virtual user 102from virtual bullets after the virtual user 102 dies in the game event514 as indicated in the virtual scene 202. The second series includes asecond set of consecutive virtual scenes in which the virtual user 102dances with other virtual users. The third series includes a third setof consecutive virtual scenes in which the virtual user 102 continues todance with the other virtual users. The second series is consecutive tothe first series and the third series is consecutive to the secondseries. The processor system 505 deletes the second and third series ofvirtual scenes and retains the first series of virtual scenes. The firstseries is the selected game event data 604.

Based on the selected game event data 604, the coaching program 530processes the metadata 606 to generate and render virtual objects, suchas overlays, that can increase game skills of the user 1 by enabling theuser 1 to see one or more reasons for occurrence of the game eventidentified by the game event identifier 636. For example, the coachingprogram 530 processes the metadata 606 to identify a position andorientation of a virtual user that shot the virtual user 102 anddetermines to include an overlay of the virtual user that shot thevirtual user 102. Once the overlay is displayed, the user 1 can see areason for the death of the virtual user 102. The reason for the deathis the virtual user that shot the virtual user 102. The overlay is anexample of a virtual object. Also, the coaching program 530 generates avirtual frame surrounding the virtual user to highlight the virtualuser. As another example, the coaching program 530 identifies, from themetadata 606, a position of the virtual bullet 106 and determines thatthe virtual user 102 would not have been killed by the virtual bullet106 if the virtual user 102 would have jumped or been protected by avirtual wall. The coaching program 530 generates a coaching comment fordisplaying to the user 1 to control the virtual user 102 to jump whileshooting or to build a wall around the virtual user 102. The coachingcomment enables the user 1 to protect the virtual user 102 from beinghit by the virtual bullet 106. The user 1 can review the coachingcomment. When faced with a similar situation in which the virtual user102 is about to be killed by another virtual bullet, if the user 1follows the coaching comment, the virtual user 102 can be protected fromthe other virtual bullet.

The coaching program 530 generates overlays of virtual objects accordingto the metadata 606 for overlaying on the selected game event data 604to generate the coaching session 608. For example, the coaching program530 overlays a virtual frame around a virtual object in a virtual scenestored as the selected event data 604. As another example, the coachingprogram 530 overlays a virtual object in the virtual scene 202. Thevirtual object overlaid in the virtual scene 202 shot the virtual bullet106 (FIG. 1). As yet another example, the coaching program 530 adds, asan overlay, a username or a user ID of the virtual object that shot thebullet 106.

The coaching program 530 provides the coaching session 608 to the clientdevice 602. For example, the processor system 505 executes the coachingprogram 530 to generate one or more coaching scenes in which one or morevirtual objects generated based on the metadata 606 are overlaid on theselected game event data 604. To illustrate, the one or more coachingscenes include an overlay of one or more virtual objects on the virtualscene 202. The one or more coaching scenes are displayed on the displayscreen of the client device 602.

The coaching session 608 continues until an input 640 is received fromthe client device 602. For example, the input 640 is generated when theuser 1 selects one or more buttons on the game controller 1 to end thecoaching session 608. The input 640 is sent within an input signal 642to the coaching program 530. The coaching program 530 identifies theinput 640 to end the coaching session 608 from the coaching program 530and ends the coaching session 608. For example, the coaching program 530ends the coaching session 530 for the user ID1 in response to thereception of the input 640. To illustrate, execution of the game program528 for the user ID1 continues after the coaching session 608 ends forthe user ID1.

In one embodiment, the coaching program 530 does not request the clientdevice 602 for identifying a game sequence type and for identifying agame event of the game sequence type. For example, the coaching program530 does not send the signals 624 and 634 to the client device 602.There is no need for the user 1 to identify a game sequence type and agame event of the game sequence type. Rather, the coaching program 530determines that the input signal 616 for initiating the coaching session608 is received immediately after the game event 514 or 518 of death ofthe virtual user 102. For example, the coaching program 530 determinesthat the input signal 616 is received at an end of occurrence of thegame event 514 or 518 and before an occurrence of a game eventconsecutive to the game event 514 are 518. Upon determining so, thecoaching program 530 determines that the coaching session 608 is to begenerated for the game event 514 or 518.

In one embodiment, the coaching session 608 occurs during a play of thegame. For example, both the coaching program 530 and the game program528 are executed simultaneously by the processor system 505. Before theuser 1 plays the game, an input signal indicating a selection forsimultaneous execution of the coaching program 530 with the game program528 is received from the game controller 1 of the client device 602 bythe processor system 505. Upon receiving the input signal, the processorsystem 505 executes the coaching program 530. When the coaching program530 is executed, one or more virtual objects are generated based on themetadata 606 and rendered by the coaching program 530 of the processorsystem 506 for display along with virtual objects of virtual scenes thatare generated by execution of the game program 528. For example, one ormore virtual objects generated based on the metadata 606 are displayedin the virtual scenes 110, 202, 302, and 402 during the coaching session608.

In one embodiment, a coaching scene is referred to herein as aself-coaching interface, such as a self-coaching user interface.

FIG. 7A is a diagram of an embodiment of a coaching scene 700 forillustrating use of one or more virtual objects with the virtual scene110 (FIG. 1). The one or more virtual objects are generated based on themetadata 606 (FIG. 6). The coaching scene 700 includes the virtual user102, the virtual gun 108, the virtual bullet 106, and the virtual ramp112. The coaching scene 700 is generated upon execution of the coachingprogram 530 (FIG. 6), and is displayed on the display device 1.

The coaching scene 700 includes a virtual comment 702, a virtual frame704, another virtual frame 706, a gun ID 708, a virtual frame 710, abullet ID 712, a gun ID 714, a virtual frame 716, a virtual gun 718, avirtual user 720, a virtual frame 722, a virtual user ID 724, a virtualgame controller 726, and a virtual button ID 728. The coaching scene 700excludes the virtual tree 105 and the virtual wall 107 so as to notclutter the coaching scene 700 with virtual objects unnecessary forcoaching the user 1. The virtual comment 702, the virtual frame 704, thevirtual frame 706, the gun ID 708, the virtual frame 710, the bullet ID712, the gun ID 714, the virtual frame 716, the virtual gun 718, thevirtual user 720, the virtual frame 722, the virtual user ID 724, thevirtual game controller 726, and the virtual button ID 728 are examplesof one or more virtual objects, such as overlays, generated based on themetadata 606. It should be noted that the virtual user 720 is not in thevirtual scene 110 (FIG. 1A).

The virtual comment 702 is a coaching comment, “Jump Now! OR Startbuilding a wall now!” for the user 1 to avoid being hit by the virtualbullet 106. The virtual frame 704 extends around the virtual user 102 tohighlight the virtual user 102. Similarly, the virtual frame 706 extendsaround the virtual gun 108. The gun ID 708 identifies a type of thevirtual gun 108 to highlight the virtual gun 108. For example, the gunID 708 identifies whether the virtual gun 108 is a shotgun or a pistolor a semiautomatic gun or a double-barrel gun or an AK-47. The virtualframe 710 extends around the virtual bullet 106 to highlight the virtualbullet 106. The bullet ID 712 identifies a type of the virtual bullet106. Examples of types of a virtual bullet include a lead round nosebullet, a semi-jacketed bullet, and a full metal jacket bullet.

The virtual frame 716 extends around the virtual gun 718 that is held bythe virtual user 720. The gun ID 714 identifies a type of the virtualgun 718. The virtual frame 722 extends around the virtual user 720 tohighlight the virtual user 720. The virtual user ID 724 is a user IDthat is assigned to another user who controls the virtual user 720 thatshot the virtual user 102 with the virtual bullet 106. The user ID 724is assigned by the processor system 505 (FIG. 5A). The virtual gamecontroller 726 is an image of the game controller 1 that is used or heldby the user 1 playing the game and controlling the virtual user 102. Thevirtual button ID 728 is a button that is selected by the user 1 duringan occurrence of the virtual scene 110 (FIG. 1) leading to a death ofthe virtual user 102. By providing the metadata 606 in the virtual scene700, the coaching program 530 facilitates self-coaching of the user 1 toincrease the skill level of the user 1.

The coaching scene 700 is displayed along or simultaneously with atimeline scrubber 730, which is a time bar or a time scale or a timeaxis or a timeline. The coaching program 530 generates and renders thetime scrubber 730, which is a portion of the coaching session 608. Theuser 1 uses the game controller 1 to select one of many segments, suchas segments 732, 734, and 736, of the timeline scrubber 732 to accessthe coaching scene 700 from the processor system 505 and view thecoaching scene 700. For example, when the segment 732 is selected, thecoaching scene 700 is displayed on the display device 1.

The timeline scrubber 730 includes segments for displaying the selectedgame event data 604 on the display screen 1. As an example, the timelinescrubber 730 includes segments and each segment can be selected by theuser 1 via the game controller 1 to generate an input signal. When theinput signals are received, a playback of the virtual scenes leading upto a game event, a virtual scene of the game event, and virtual scenesoccurring after the game event are played back by the coaching program530 with overlay content superimposed on one or more of the virtualscenes. As another example, the timeline scrubber 730 includes a set 738of segments that can be selected by the user 1 via the game controller 1to enable a display of coaching scenes in which one or more virtualobjects generated based on the metadata 606 are overlaid on portions ofvirtual scenes that are recorded during the predetermined time periodbefore the time at which a virtual scene, such as the virtual scene 202,of death of the virtual user 102 is recorded. The predetermined timeperiod extends from −P time units to 0 time units, where time units canbe seconds or minutes. Moreover, the timeline scrubber 730 includes asegment 736 that can be selected by the user 1 via the game controller 1to view a coaching scene in which one or more virtual objects generatedbased on the metadata 606 are overlaid on a portion of a virtual scene,such as the virtual scene 202, that is recorded at a time of death ofthe virtual user 102. When the segment 736 of the timeline scrubber 730is selected by the user 1 via the game controller 1, the coaching sceneincluding one or more virtual objects generated based on the metadata606 are overlaid on one or more of the virtual objects in the virtualscene 202 in which the virtual user 102 died is rendered by the coachingprogram 530 and displayed on the display device 1. The death of thevirtual user 102 is recorded at 0 units. The timeline scrubber 730includes a set 740 of segments that can be selected by the user 1 viathe game controller 1 to enable a display of coaching scenes in whichone or more virtual objects generated based on the metadata 606 areoverlaid on portions of virtual scenes that are recorded during thepreset time period after the time at which a virtual scene, such as thevirtual scene 202, of death of the virtual user 102 is recorded. Thepreset time period extends from 0 time units to M time units, where timeunits can be seconds or minutes. There is a window of time between thetime units −P and M.

In one embodiment, the coaching scene 700 excludes one or more of thevirtual frame 704, the virtual frame 706, and the gun ID 708. Forexample, the coaching program 530 determines that one or more of thevirtual frame 704, the virtual frame 706, and the gun ID 708 is notneeded to increase the skill level of the user 1, and therefore does notgenerate one or more of the virtual frame 704, the virtual frame 706,and the gun ID 708.

In an embodiment, instead of extending a frame around a virtual objectin a coaching scene, the coaching program 530 highlights the virtualobject by rendering the virtual object in a substantially differentcolor or intensity or shade compared to colors or intensities are shadesof other virtual objects in the coaching scene. For example, instead ofextending the frame 710 around the virtual bullet 106 and the frame 722around the virtual user 720, the coaching program 530 renders thevirtual bullet 106 and the virtual user 720 to have a substantiallygreater intensity than intensities of the virtual user 102 and thevirtual gun 108. Again, highlighting the virtual bullet 106 and thevirtual user 720 in comparison with the virtual user 102 and the virtualgun 108 facilitates self-coaching of the user 1.

In an embodiment, the coaching scene 700 includes the virtual tree 105and the virtual wall 107.

In one embodiment, the coaching program 530 changes a time periodbetween the time units −P and M based on a number of game events thatlead up to an end game event, such as death or decrease in health level.For example, the coaching program 520 determines that a first length oftime for which the virtual user 102 engages in a battle sequence thatleads to a first death of the virtual user 102 is greater than a secondlength of time for which the virtual user 102 engages in a second battlesequence that leads to a second death of the virtual user 102. The firstbattle sequence is fought with one or more virtual users and the secondbattle sequence is fought with one or more virtual users. The coachingprogram 520 stores the selected game event data 604 that leads up to thefirst death for the first length of time and stores the selected gameevent data 604 that leads up to the second death for the second lengthof time. In this manner, the entire first and second battle sequencesare captured and stored by the coaching program 520.

FIG. 7B is a diagram of an embodiment of a coaching scene 750 that isrendered and displayed on the display device 1. The coaching scene 750is rendered by the coaching scene 750 in response to an input signalindicating a selection of a segment 752 on the timeline scrubber 730.The segment 752 is selected by the user 1 using one or more buttons onthe game controller 1.

The segment 752 is consecutive to the segment 734 and the segment 734 isconsecutive to the segment 732. For example, the segment 752 includesone or more virtual objects from a virtual scene that is recordedbetween a time at which the virtual scene 110 (FIG. 1) is recorded and atime at which the virtual scene 202 (FIG. 2) is recorded. Each virtualscene corresponding to a segment of the timeline scrubber 730 isrecorded after being displayed on the display screen 1.

The coaching scene 750 includes a virtual comment 754, a virtual frame756, a virtual bullet 758, and a bullet ID 760. The virtual comment 754is generated by the coaching program 530 (FIG. 6) to coach the user 1 toprotect the virtual user 102 from being shot by the virtual bullet 758.For example, the virtual comment 754 includes “Jump Now!”. The virtualframe 756 extends around or surrounds the virtual bullet 758 that isshot by the virtual user 720 after the virtual bullet 106 (FIG. 7A) isshot. The virtual frame 756 highlights the virtual bullet 758. Thebullet ID 760 identifies a type of the virtual bullet 758. The coachingscene 750 excludes the virtual tree 105 and the virtual wall 107 so asto not clutter the coaching scene 750 with virtual objects unnecessaryfor coaching the user 1.

It should be noted that in the coaching scene 750, a progression ofvirtual objects compared to the virtual objects in the coaching scene700 of FIG. 7A is illustrated. For example, in the coaching scene 750,the virtual frame 756 that surrounds the virtual bullet 758 shot fromthe virtual gun 718 is illustrated. The virtual frame 756 is generatedafter the virtual frame 710 that surrounds the virtual bullet 106 isgenerated. The virtual bullet 758 is shot after the virtual bullet 106is shot. Also, in the coaching scene 750, the virtual comment 754indicates to the user 1 to control the virtual user 102 to jump withoutan option for building a virtual wall. The coaching program 530determines that at the time the virtual bullet 758 is shot, it is toolate for the user 1 to build the virtual wall.

In one embodiment, when a segment 753 next to the segment 752 isselected by the user 1 via the game controller 1, the processor system505 processes the metadata 606 to display a position of the virtualbullet 758 that is closer to the virtual user 102 compared to a positionof the virtual bullet 758 illustrated in the coaching scene 750. Thesegment 753 corresponds to a coaching scene that includes one or morevirtual objects from a first virtual scene that is consecutive in timefrom a time at which a second virtual scene is recorded by the gamerecorder 502. The positions of the virtual bullet 758 in the first andsecond virtual scenes define movement of the virtual bullet 758. Thecoaching scene 750 includes virtual objects from a playback of thesecond virtual scene.

In an embodiment, the coaching scene 750 includes the virtual tree 105and the virtual wall 107.

In one embodiment, the virtual comments 702 and 754 are examples ofhints to the user 1 to increase the skill level to be above the presetthreshold. For example, during a next gaming session, when the user 1controls the virtual user 102 or build a virtual wall on at least oneside of the virtual user 102, chances of the virtual user 102 from beinghit by a virtual bullet from a side or behind the virtual user 102 arereduced. The virtual user 102 can stay alive longer during the gamesession and the game program 528 increases the skill level of the user 1to be above the preset threshold. Each hint provides a reason for adeath of the virtual user 102. For example, the virtual user 102 diedduring a previous gaming session because the virtual user 102 did notbuild a virtual wall or did not jump during the previous gaming session.

FIG. 8A is a diagram of an embodiment to illustrate a simultaneousdisplay of the coaching scene 700 and another coaching scene 800 on thedisplay device 1. In addition to the coaching scene 700, the coachingscene 800 is rendered by the coaching program 530 (FIG. 6) on thedisplay device 1. For example, the coaching session 608 (FIG. 6)includes the coaching scenes 700 and 800. As another example, an inputsignal is received by the coaching program 530 from the client device602. The input signal indicates a selection of the game event identifier636 (FIG. 6) of game event data recorded from the virtual scene 110(FIG. 1) based on which the coaching scene 700 is generated and aselection of another game event identifier of game event data recordedfrom which the virtual scene 302 (FIG. 3) based on which the coachingscene 800 is generated. In this example, when a selection of the segment732 on the timeline scrubber 730 is received from the client device 602,both the coaching scenes 700 and 800 are generated and rendered by thecoaching program 530. Also in this example, the timeline scrubber 730 isgenerated by the coaching program 530 and rendered along the coachingscenes 700 and 800.

The coaching scene 800 is generated based on the virtual scene 302 (FIG.3). For example, the virtual scene 302 is recorded and one or morevirtual objects from the virtual scene 302 are included in the coachingscene 800. To illustrate, the virtual user 102, the virtual bullet 306,and the virtual gun 108 are included in the coaching scene 800. Asanother example, the coaching scene 800 is generated based on thevirtual scene 302 that is displayed and recorded at a time correspondingto the segment 732. To illustrate, the virtual scene 302 is recorded anumber of time segments prior to a time of the game event q of thevirtual user 102. The virtual scene 302 is recorded in the same manneras that of the virtual scene 110 from which the coaching scene 700 isgenerated.

The coaching scene 800 includes a user ID 802, a virtual frame 804, avirtual user 806, a virtual gun 808, a virtual frame 810, a gun ID 812,the virtual bullet 306, a bullet ID 816, and a virtual frame 818. Theuser ID 802, the virtual frame 804, the virtual user 806, the virtualgun 808, the virtual frame 810, the gun ID 812, the bullet ID 816, andthe virtual frame 818 are examples of one or more overlays of virtualobjects generated based on the metadata 606 (FIG. 6).

The virtual frame 804 extends around the virtual user 806 to highlightthe virtual user 806 that is a reason for the death x of the virtualuser 102. The virtual user 806 is controlled by another user via a gamecontroller. The virtual frame 810 extends around the virtual gun 808held by the virtual user 806 to highlight the virtual gun 808. The gunID 812 identifies a type of the virtual gun 808. The user ID 802 isassigned to the other user that controls the virtual user 806. The userID 802 is assigned by the processor system 505 (FIG. 5A).

The virtual frame 818 extends around the virtual bullet 306 to highlightthe virtual bullet 306 that is directed towards the virtual user 102 bythe virtual user 806. The bullet ID 816 identifies the virtual bullet306. Also, the coaching scene 800 includes the virtual comment 702 tofacilitate coaching of the user 1.

In one embodiment, a virtual comment displayed within the coaching scene800 is different from the virtual comment 702. For example, the virtualcomment displayed within the coaching scene 800 is “Jump!” or “Build awall” instead of “Jump now! OR Start building a wall now!”.

In an embodiment, one or more virtual objects in the coaching scene 800are not highlighted by the coaching program 530. For example, thecoaching program 530 determines that there is no need for highlightingthe virtual user 102 and the virtual gun 108 to increase the skill levelof the user 1, and determines not to generate the virtual frame 704 andthe virtual frame 706.

FIG. 8B is a diagram of an embodiment of the display device 1 toillustrate a simultaneous rendering and display of the coaching scene700 associated with the user ID1 and another coaching scene 850associated with the user ID N. The coaching scene 850 is generated bythe coaching program 530 (FIG. 5A). The coaching scene 850 is generatedfor the user ID N. For example, when the input signal 623 (FIG. 6)includes the user ID1 identifying the user 1 and includes the user ID Nidentifying another user who is assigned a user ID N, the coachingprogram 530 accesses the game event data 504 recorded for the user ID Nfrom one or more memory devices of the processor system 505 and stores aportion of the game event data 504 as a portion of the selected gameevent data 604 in one or more memory devices of the processor system505. To illustrate, the one or more memory devices in which the gameevent data 504 is recorded are different than the one or more memorydevices in which the selected game event data 604 is recorded. The oneor more virtual objects generated based on the metadata 606 for the userID N are overlaid by the coaching program 530 on the portion of theselected game event data 604 for the user ID N to generate the coachingscene 850 for the user ID N. The game event data 504 is generated duringgame play of the game by the other user having the user ID N.

The coaching scene 850 includes a virtual tree 852, a virtual user 854,a virtual gun 856, a gun ID 858, a user ID 860, a virtual frame 862,another virtual frame 864, a virtual user 866, a virtual gun 868, avirtual frame 870, another virtual frame 872, a gun ID 874, a virtualbullet 876, a virtual frame 878, a gun ID 880, a virtual user 882, avirtual frame 884, a virtual gun 886, a virtual frame 888, a virtualwall 890, another virtual wall 892, a virtual health 894, a user ID 897,and a virtual ramp 899. The coaching scene 850 further includes avirtual controller 896, and a virtual button 898. The user ID 860, thevirtual frame 862, the virtual frame 864, the virtual user 854, thevirtual gun 856, the gun ID 858, the virtual frame 872, the gun ID 874,the virtual frame 888, the virtual frame 884, the virtual frame 870, thevirtual controller 896, and the virtual button 898 are examples of oneor more virtual objects generated based on the metadata 606 for overlay.

The coaching scene 850 is generated for the user ID N based on a virtualscene that is generated upon execution of the game program 528 (FIG.5A), and leads up to a death of the virtual user 882. For example, thevirtual scene from which the coaching scene 850 is generated includesone or more of virtual objects displayed in the coaching scene 850. Thevirtual scene from which the coaching scene 850 is generated excludesone or more virtual objects generated based on the metadata 606 fordisplay in the coaching scene 850. To illustrate, the virtual scene fromwhich the coaching scene 850 is generated includes the virtual bullet876, the virtual gun 886, the virtual user 882, the virtual ramp 899,the virtual walls 890 and 892, the virtual tree 852, the user ID 897,the virtual user 866, and the virtual gun 868. The virtual scene for theuser ID N is generated after the other user accesses the game program528. The game program 528 is accessed by the other user when the user IDN is authenticated by the processor system 505.

The virtual frame 862 surrounds the virtual user 854 and the virtualframe 864 surrounds the virtual gun 856. The gun ID 858 identifies atype of the virtual gun 856. The user ID 860 is assigned to a user thatcontrols the virtual user 854 via a game controller or another type ofcontroller. The virtual frame 878 extends around the virtual bullet 876that is shot from the virtual gun 856 towards the virtual user 882.

The virtual health 894 is a health of the virtual user 882. The virtualframe 884 extends around the virtual user 882 and the virtual frame 888extends around the virtual gun 886. The virtual user 882 is standing onthe virtual ramp 899 and is surrounded on two sides by the virtual walls890 and 892. The gun ID 880 identifies a type of the virtual gun 886.Also, the gun ID 874 identifies a type of the virtual gun 868. Thevirtual frame 872 surrounds the virtual gun 868 and the virtual frame870 surrounds the virtual user 866. The user ID 897 identifies a userwho is controlling the virtual user 866 via a game controller or anothertype of controller.

The virtual controller 896 is an image of a controller, such as akeyboard, that is used by the other user to control the virtual user882. The virtual button 898 identifies a button on the controller thatis represented by the virtual controller 896. The button is selected bythe other user at the time corresponding to the segment 732 during theplay of the game.

The timeline scrubber 730 is displayed along or simultaneous with adisplay of the coaching scenes 700 and 850. For example, the coachingprogram 530 (FIG. 6) renders the timeline scrubber 730 along with thesimultaneous display of the coaching scenes 700 and 850, for display ofthe timeline scrubber 730 with the coaching scenes 700 and 850. When aselection of the segment 732 is received from the game controller 1(FIG. 1) of the client device 602 (FIG. 6), the coaching program 530renders the coaching scene 850, which is generated based on a virtualscene that is recorded at a time corresponding to the segment 732. Thesegment 736 indicates the time 0 at which a death of the virtual user882 occurs during a play of the game by the other user that controls thevirtual user 882.

The virtual user 882 is shooting at the virtual user 866. During theshootout, the virtual user 854 is shooting at the virtual user 882. Thevirtual wall 892 protects the virtual user 882 from being injured by thevirtual bullet 876. The user 1 can learn from the coaching scene 850 tobuild a virtual wall, such as the virtual wall 890, on a side of thevirtual user 102 to protect the virtual user 102 from the virtual bullet106 shot by the virtual user 720.

In one embodiment, instead of the coaching program 530, a coachingengine is used. As an example, an engine is a combination of softwareand hardware for executing functions described herein as being performedby the engine. To illustrate, the engine is a PLD or an ASIC that isprogrammed to perform the functions described herein as being performedby the engine.

In one embodiment, any virtual frame, described herein, surrounds avirtual object to highlight the virtual object.

FIG. 8C is a diagram of an embodiment of a system 801 to illustrate thatinstead of or in addition to rendering a coaching scene on the displayscreen 1, the processor system 505 generates audio data 805 that isoutput as sound to the user 1 via a head phone 803. The head phone 803is worn to by the user 1 to be proximate to ears of the user 1. Duringthe coaching session 608, the coaching program 530 generates the audiodata 805 that is output as sound to the user 1. For example, during thecoaching session 608, instead of or in addition to displaying thevirtual comment 702, the coaching program 530 sends the audio data 805to the headphone 803. The audio data 805 includes a message, such as“Jump while shooting”, that is output as sound simultaneously with orinstead of the virtual comment 702 displayed on the display screen 1.The headphone 803 outputs the audio data 805 as the message to the user1.

In an embodiment, the virtual objects 702, 704, 706, 708, 710, 712, 714,716, 718, 720, 722, 724, 726, 728 (FIG. 7A), 756, 758 (FIG. 7B), 802,804, 806, 808, 810, 812, 816, 818 (FIG. 8A), 854, 856, 858, 860, 864,878, 880, 884, 888, 870, 872, 874, and 897 are examples of overlaycontent that is generated by the coaching program 530 based on themetadata 606.

FIG. 9 is a diagram of an embodiment of a system 900 to illustrate useof an inferred training engine 902 for generating the coaching session608 (FIG. 6). The system 900 includes an artificial intelligence (AI)processor 904, the game event data 510 for the user ID1, and the gameevent data 504 for the user IDs 2-N. The AI processor 904 is a processorof the processor system 505 (FIG. 5A).

The inferred training engine 902 includes a feature extractor 906, afeature classifier 908, and a model 910 that is to be trained. Anexample of each of the feature extractor 906, the feature classifier908, and the model 910 is an ASIC. Another example of each of thefeature extractor 906, the feature classifier 908, and the model 910 isa PLD. An example of the model 910 is a network of circuit elements.Each circuit element has one or more inputs and one or more outputs. Aninput of a circuit element is coupled to one or more outputs of one ormore circuit elements. To illustrate, the model 910 is a neural networkor an artificial intelligence model. The feature classifier 908 iscoupled to the feature extractor 906 and to the model 910.

The inferred training engine 902 accesses, such as reads, the game eventdata 510 and the game event data 504 from one or more memory devices ofthe game recorder 502. The feature extractor 906 extracts features fromthe game event data 510 and 504. Once the features are extracted, thefeature classifier 908 classifies the features that are extracted. Thefeatures that are classified are used to train the model 910 todetermine a game event, such as death or decrease in health level, forwhich to initiate the coaching session 608 and to identify one or morevirtual objects of a virtual scene that are to be associated with one ormore overlays of virtual objects generated based on the metadata 606.

FIG. 10 is a flowchart to illustrate an embodiment of a method 1000 fortraining the model 910. In an operation 1002 of the method 1000, thefeature extractor 906 identifies features from the game event data 510and the game event data 504 (FIG. 9). For example, the feature extractor906 determines that a skeletal of the virtual user 102 in the virtualscene 202 (FIG. 2) lies in a horizontal plane instead of a verticalplane, or determines that the virtual scene 202 includes the virtualdrone 204 above the skeletal of the virtual user 102 for beaming thevirtual user 102, or determines that words, such as, “I am going to die”or “I am dead” are uttered by the virtual user 102 in the virtual scene202, or a combination thereof to determine a death of the virtual user102 in the virtual scene 202. As another example, the feature extractor906 determines that a health level of the virtual health of the virtualuser 102 in the virtual scene 202 has decreased to zero to determinethat the health level has decreased to be below the predeterminedthreshold.

An operation 1004 of the method 1000 occurs after the operation 1002. Inthe operation 1004, the feature classifier 908 classifies the featuresextracted from the game event data 510 and 504. For example, the featureclassifier 908 determines that because the skeletal of the virtual user102 lies in the horizontal plane, the virtual user 102 has died. Asanother example, the feature classifier 908 determines that because thevirtual scene 202 includes the virtual drone 204 above the skeletal ofthe virtual user 102, the virtual user 102 has died. As yet anotherexample, the feature classifier 908 determines that because the virtualscene 202 includes the word “I” and “dead” or “I” and “die” in the samesentence uttered by the virtual user 102 in the virtual scene 202, thevirtual user 102 has died. As another example, the feature classifier908 determines that because the health level of the virtual user 102 inthe virtual scene 202 has decreased to be below the predeterminedthreshold, health of the virtual user 102 is low.

An operation 1006 of the method 1000 occurs after the operation 1004. Inthe operation 1006, the model 910 is trained based on a number of gameevents of a sequence type to determine a probability of occurrence of agame event of the sequence type during a next gaming session. Forexample, the model 910 determines a probability that the virtual user102 will die during a next gaming session or a probability that a healthlevel of the virtual user 102 will decrease below the threshold during anext gaming session. To illustrate, upon determining that the virtualuser 102 has died during at least 6 out of the past 10 gaming sessions,the model 910 determines that a probability that the virtual user 102will die during a next gaming session is high, e.g., above a presetthreshold. On the other hand, upon determining that the virtual user 102has survived during at least 6 out of the past 10 gaming sessions, themodel 910 determines that a probability that the virtual user 102 willdie during a next gaming session is low, e.g., below the presetthreshold.

An operation 1008 of the method 1000 follows the operation 1006. Theoperation 1008 is executed by the processor system 505. Upon determiningthat the probability of occurrence of a game event of a sequence typeduring the next gaming session is low, the processor system 505 does notrecommend, in the operation 1008, that the coaching session 608 beinitiated for the next gaming session. On the other hand, upondetermining that the probability of occurrence of a game event of asequence type during the next gaming session is high, the processorsystem 505 recommends, in the operation 1008, that the coaching session608 be initiated for the next gaming session. For example, the processorsystem 505 generates a message and renders the message for display onthe display device 1. The message queries the user 1 whether the user 1wishes to initiate the coaching session 608. As another example, theprocessor system 505 generates audio data including the message toindicate to the user 1 whether the user 1 wishes to initiate thecoaching session 608, and sends the audio data to the head phone 910(FIG. 9). The audio data is output as sound by the head phone 910 to theuser 1.

Upon viewing the message displayed on the display screen 1 or listeningto the message, which is output as sound by the head phone 910, the user1 selects one or more buttons on the game controller 1 to indicate thatthe coaching session 608 be initiated to generate an input signal. Uponreceiving the input signal indicating that the coaching session 608 beinitiated, the processor system 505 initiates the coaching session 608.For example, the processor system 505 renders the coaching scene 700(FIG. 7A) or the coaching scene 750 (FIG. 7B) or both the coachingscenes 700 and 800 (FIG. 8A) or both the coaching scenes 700 and 850(FIG. 8B) for display on the display screen 1. On the other hand, uponreceiving an input signal indicating that the coaching session 608 notbe initiated, the processor system 505 does not initiate the coachingsession 608.

FIG. 11A is a diagram of an embodiment of a system 1100 to illustrate acommunication via a router and modem 1104 and a computer network 1102between the processor system 505 and multiple devices, which include adisplay device 1106 and a hand-held controller 1108. Examples of thedisplay device 1106 include the display device 1 (FIG. 1), an LCDdisplay device, an LED display device, a plasma display device, and anHMD. Examples of the hand-held controller 1108 include a touch screen, akeypad, a mouse, the game controller 1, and a keyboard. Each of thedisplay device 1106 and the hand-held controller 1108 is used by theuser 1. Also, each of the users 2-N uses a hand-held controller havingthe same structure and function as that of the hand-held controller 1108and a display device having the same structure and function as that ofthe display device 1106. The display device 1106 and the hand-heldcontroller 1108 are examples of the client device 602 (FIG. 6).

The system 1100 further includes the router and modem 1104, the computernetwork 1102, and the processor system 505. The system 1100 alsoincludes a headphone 1110 and the display device 1106. The displaydevice 1106 includes a display screen 1112, such as an LCD displayscreen, and LED display screen, or a plasma display screen. The displaydevice 1 (FIG. 1) is an example of the display device 1106. An exampleof the computer network 1102 includes the Internet or an intranet or acombination thereof. An example of the router and modem 1104 includes agateway device. Another example of the router and modem 1104 includes arouter device and a modem device.

The display screen 1112 is coupled to the router and modem 1104 via awired connection. Examples of a wired connection, as used herein,include a transfer cable, which transfers data in a serial manner, or ina parallel manner, or by applying a universal serial bus (USB) protocol.

The hand-held controller 1108 includes controls 1118, a digital signalprocessor system (DSPS) 1120, and a communication device 1122. Thecontrols 1118 are coupled to the DSPS 1120, which is coupled to thecommunication device 1122. Examples of the controls 1118 include buttonsand joysticks. Examples of the communication device 1122 includes acommunication circuit that enables communication using a wirelessprotocol, such as Wi-Fi™ or Bluetooth™, between the communication device1122 and the router and modem 1104.

The communication device 1122 is coupled to the headphone 1110 via awired connection or a wireless connection. Examples of a wirelessconnection, as used herein, include a connection that applies a wirelessprotocol, such as a Wi-Fi™ or Bluetooth™ protocol. Also, thecommunication device 1122 is coupled to the router and modem 1104 via awireless connection. Examples of a wireless connection include a Wi-Fi™connection and a Bluetooth™ connection. The router and modem 1104 iscoupled to the computer network 1102, which is coupled to the processorsystem 505.

During the play of the game, the processor system 505 executes the gameprogram 528 to generate image frame data from one or more game states ofthe game and applies a network communication protocol, such as transfercontrol protocol over Internet protocol (TCP/IP), to the image framedata to generate one or more packets and sends the packets via thecomputer network 1102 to the router and modem 1104. The modem of therouter and modem 1104 applies the network communication protocol to theone or more packets received from the computer network 1102 to obtain orextract the image frame data, and provides the image frame data to therouter of the router and modem 1104. The router routes the image framedata via the wired connection between the router and the display screen1112 to the display screen 1112 for display of one or more images of thegame based on the image frame data received within the one or morepackets.

During the display of one or more images of the game, the game recorder502 records the game event data 510 and 504, which is used to generatethe coaching session 608 or train the model 910 or a combinationthereof. Also, during execution of the game program 528, the controls1118 of the hand-held controller 1108 are selected or moved by the user1 to generate an input signal, such as the input signal 616, or 623, or630, or 638, or 642, or an input signal indicating a selection of asegment of the timeline scrubber 730, or any other input signaldescribed herein, which is processed by the DSPS 1120. The DSPS 1120processes, such as measures or samples or filters or amplifies or acombination thereof, the input signal to output a processed inputsignal, which has the same information as that within the input signal.For example, the DSPS 1120 identifies a button of the game controller 1selected by the user 1. As another example, the DSPS 1120 identifieswhether a joystick of the game controller 1 is moved or a button of thegame controller 1 is selected by the user 1. The processed input signalis sent from the DSPS 1120 to the communication device 1122. Thecommunication device 1122 applies the wireless protocol to the processedinput signal to generate one or more wireless packets and sends thewireless packets to the router and modem 1104. The wireless packetsinclude the same information as that included within the processed inputsignal.

The router of the router and modem 1104 receives the wireless packetsfrom the communication device 1122, and applies the wireless protocol toobtain or extract the processed input signal from the wireless packets.The router of the router and modem 604 provides the processed inputsignal to the modem of the router and modem 604. The modem applies thenetwork communication protocol to the processed input signal and togenerate one or more network packets. For example, the modem determinesthat the processed input signal is to be sent to the processor system502 that is executing the game program 506, and embeds a network addressof the processor system 505 within the one or more network packets. Themodem sends the one or more network packets via the computer network1102 to the processor system 505.

The processor system 505 applies the network communication protocol tothe one or more network packets received from the router and modem 1104to obtain or extract the information within the processed input signal,and processes the information in a manner explained above with referenceto FIG. 6 to initiate the coaching session 608 or to recommendinitiation of the coaching session 608 (FIG. 6). The processor system505 generates an output signal, such as the output signal 618, or 624,or 634 (FIG. 6), or any other output signal, described herein, or anyother output signal including image frame data of a coaching scene,described herein, or an output signal including audio data, describedherein, and applies the network communication protocol to the outputsignal to generate one or more network packets. The processor system 505sends the one or more network packets via the computer network 1102 tothe router and modem 1104.

The modem of the router and modem 1104 applies the network communicationprotocol to the one or more network packets received via the computernetwork 1102 to obtain or extract the output signal. The router of therouter and modem 1104 applies the wireless protocol to the output signalto generate one or more wireless packets and sends the wireless packetsto the communication device 1122 of the hand-held controller 1108. Thecommunication device 1122 of the hand-held controller 1108 applies thewireless protocol to the one or more wireless packets received from therouter and modem 1104 to obtain or extract the output signal and sendsthe output signal to the headphone 1110 for output of the audio data assound to the user 1. For example, the communication device 1122 appliesa wired protocol, such as a universal serial bus (USB) protocol, togenerate one or more packets having the audio data and sends the one ormore packets via the wired connection to the headphone 1110. As anotherexample, the communication device 1122 applies the wireless protocol tothe audio data to generate one or more wireless packets and sends theone or more wireless packets via the wireless connection to theheadphone 1110.

Also, the router of the router and modem 1104 sends the output signal tothe display screen 1112. Upon receiving the output signal, the displayscreen 1112 displays one or more image frames according to the imageframe data of the output signal.

In one embodiment, the communication device 1122 communicates with therouter and modem 1104 via a wired connection, such as a cable.

In one embodiment, the display screen 1112 is coupled to the router andmodem 1104 via a communication device, such as a communication devicethat applies a wireless communication protocol. The communication deviceis a part of the display device 1106. For example, the display screen1112 is coupled to the communication device. The router 1104 applies thewireless protocol to the image frame data received via the computernetwork 1102 to generate one or more wireless packets and sends the oneor more wireless packets to the communication device of the displaydevice 1106. The communication device applies the wireless protocol tothe one or more wireless packets to extract or obtain the image framedata and sends image frame data to the display screen 1112 for displayof one or more images of a coaching scene, described herein.

In one embodiment, the display device 1106 and the hand-held controller1108 are integrated within a mobile device, such as a smartphone or atablet or a laptop.

FIG. 11B is a diagram of an embodiment of a system 1120 to illustratethat the processor system 505 can be implemented within a game console1122. The system 1120 includes the display device 1106, the hand-heldcontroller 1108, the head phone 1110, the game console 1122, the routerand modem 1104, the computer network 1102, and a server system 1124. Anexample of the game console 1124 is a video game console or a computeror a combination of a central processing unit (CPU) and a graphicsprocessing unit (GPU). To illustrate, the game console 1124 is a SonyPlayStation™ or a Microsoft Xbox™. The game console 1124 includes theprocessor system 505 and a communication device 1126, such as Wi-Fi™communication device or a Bluetooth™ communication device. As anexample, a processor system as used herein, includes one or more CPUsand one or more GPUs, and the one or more CPUs are coupled to the one ormore GPUs.

An example of the server system 1124 includes one or more servers withinone or more data centers. To illustrate, a server can be a game consoleor a server blade. As another example, the server system 1124 includesone or more virtual machines. The communication device 1126 is coupledto the communication device of the display device 1106 via a wirelessconnection, such as a Wi-Fi™ connection or a Bluetooth™ connection.Moreover, the communication device 1126 is coupled to the communicationdevice 1122 of the hand-held controller 1108 via a wireless connection.The communication device 1126 is coupled to the processor system 505.The processor system 505 is coupled to the router and modem 1104 via awired connection. The router and modem 1104 is coupled via the computernetwork 1102 to the server system 1124.

The processor system 505 instead of or in conjunction with the serversystem 1124 executes the game for display of virtual scenes on thedisplay screen 1112 of the display device 1106. For example, in responseto receiving login information that is provided by the user 1 via thegame controller 1, the processor system 505 sends a request to theserver system 1124 via the computer network 1102 to determine whetherthe login information is valid. Upon receiving an indication from theserver system 1124 via the computer network 1102 that the logininformation received from the game controller 1 is valid, the processorsystem 505 executes the game program 528 for play of the game by theuser 1 via the game controller 1 and the game console 1122. On the otherhand, upon receiving an indication from the server system 1124 via thecomputer network 1102 that the login information received from the gamecontroller 1 is invalid, the processor system 505 does not execute thegame program 528 for play by the user 1 via the game controller 1 andthe game console 1122.

The communication device 1126 receives the wireless packets having theinput signal, such as the input signal 616, or 623, or 630, or 638, or642 (FIG. 6), or an input signal indicating a selection of a segment ofthe timeline scrubber 730, or any other input signal described herein,from the hand-held controller 1108, and applies the wireless protocol tothe wireless packets to extract the input signal from the wirelesspackets, and provides the input signal to the processor system 505. Theprocessor system 505 generates an output signal, such as the outputsignal 618, or 624, or 634 (FIG. 6), or any other output signal,described herein, or any other output signal including image frame dataof a coaching scene, described herein, or an output signal includingaudio data, described herein, based on the input signal in a mannerdescribed above. For example, the processor system 505 generates datafor the coaching session 608. The processor system 505 provides theoutput signal to the communication device 1126. The communication device1126 applies the wireless protocol to the output signal to generate oneor more wireless packets and sends the wireless packets to thecommunication device 1122 of the hand-held controller 1108 or to thedisplay device 1106, or to both the display device 1106 and thehand-held controller 1108.

In one embodiment, some of the functions described herein as beingperformed by the processor system 505 are performed by the processorsystem 505 of the game console 1122 and the remaining functions,described herein as being performed by the processor system 505, areinstead performed by the server system 1124.

In an embodiment, the processor system 505 is coupled to the displaydevice 1106 via a wired connection. The output signal is sent from theprocessor system via the wired connection to the display device 1106 fordisplay of one or more images of a coaching scene, described herein, ora virtual scene, described herein, on the display device 1106.

In one embodiment, the game is stored on the game console 1122 and is anon-networked game. In this embodiment, the game is a single player gameor a multi-player game and the game console 1122 does not communicate tothe server system 1124 to access any portion of the game from the serversystem 1124.

FIG. 11C is a diagram of an embodiment of a system 1140 to illustratecommunication between a smart phone 1142 and the processor system 505via the computer network 1102 without using the router and modem 1104between the computer network 1102 and the smart phone 1142. The system1140 includes the smart phone 1142, the head phone 1110, a cellularnetwork 1144, the computer network 1102, and the processor system 505.The smart phone 1142 is an example of the client device 602 (FIG. 6).

The smart phone 1142 is coupled to the cellular network 1144 via acellular wireless connection, such as a fourth-generation cellularwireless (4G) connection or a fifth cellular wireless (5G) connection.The cellular network 1144 is coupled to the computer network 1102, whichis coupled to the processor system 505.

The smart phone 1142 generates one or more packets by applying acellular communication protocol, such as the 4G or the 5G protocol, tothe input signal, such as the input signal 616, or 623, or 630, or 638,or 642, or an input signal indicating a selection of a segment of thetimeline scrubber 730, or any other input signal described herein, andsends the one or more packets to the cellular network 1144. The cellularnetwork 1144 receives the one or more packets and applies the cellularcommunication protocol to obtain or extract the input signal, andapplies the network communication protocol to the input signal togenerate one or more network packets. The one or more network packetsgenerated by the cellular network 1144 are sent via the computer network1102 to the processor system 505. The processor system 505 processes theone or more network packets received from the cellular network 1102 in amanner described above to generate an output signal, such as the outputsignal 618, or 624, or 634, or any other output signal, describedherein, or any other output signal including image frame data of acoaching scene, described herein, or an output signal including audiodata, described herein, and sends one or more network packets includingthe output signal via the computer network 1102 to the cellular network1144.

The cellular network 1144 applies the network communication protocol tothe one or more network packets received from the processor system 505to extract or obtain the output signal, and applies the cellularcommunication protocol to the output signal to generate one or morepackets. The cellular network 1102 sends the one or more packetsincluding the output signal to the smart phone 1142.

FIG. 12A is a diagram of an embodiment of the headphone 1110. Theheadphone 1110 includes a communication device 1202, a digital-to-analog(D/A) converter 1204, an audio amplifier 1206, and a speaker 1208. Anexample of the communication device 1202 is a communication circuit thatapplies the wired protocol or the wireless protocol.

The communication device 1202 is coupled to the communication device1122 (FIG. 11A) of the hand-held controller 1108 or to the smart phone1142 (FIG. 11C). The digital-to-analog converter 1204 is coupled to thecommunication device 1202 and the audio amplifier 1206 is coupled to thedigital-to-analog converter 1204. Also, the speaker 1208 is coupled tothe audio amplifier 1206.

The communication device 1202 receives one or more packets having theaudio data 805 (FIG. 8C) from the communication device 1122 (FIG. 11A)or from the smart phone 1142 (FIG. 11C), and applies a protocol, such asthe wired protocol or the wireless protocol, to extract or obtain theaudio data 805 from the one or more packets. The communication device1202 sends the audio data 805 to the digital-to-analog converter 1204.The digital-to-analog converter 1204 converts the audio data 805 from adigital format to an analog format to output analog audio signals. Thedigital-to-analog converter 1204 sends the analog audio signals outputbased on the audio data 805 to the audio amplifier 1206. The audioamplifier 1206 amplifies, such as increases an amplitude or a magnitude,of the analog audio signals to output amplified audio signals, which areelectrical signals. The speaker 1208 converts electrical energy of theamplified audio signals into sound energy to output sounds to be heardby the user 1 (FIG. 1).

In one embodiment, instead of the speaker 1208, multiple speakers areused.

FIG. 12B is a diagram of an embodiment of the display device 1250 to fordisplaying an image of a coaching scene or a virtual scene, describedherein, on the display screen 1112. The display device 1250 is anexample of the display device 1 (FIG. 1). The display device 1250includes a communication device 1252 and the display screen 1112.Examples of the display device 1250 include an LCD display device, anLED display device, and a plasma display device. Examples of the displayscreen 1112 include an LCD display screen, and LED display screen, and aplasma display screen. To illustrate, the display device 1250 is adisplay device of the smart phone 1142 or of the game controller 1 or ofa tablet or of a computer. Examples of a computer include a desktopcomputer and a laptop computer. Examples of the communication device1252 include a communication circuit that applies the wired or wirelessprotocol for communication of data. The communication device 1252 iscoupled to the display screen 1112.

Instead of or in addition to generating other forms of data, such as theaudio data 805 (FIG. 8C), the processor system 505 generates image framedata of a coaching scene or a virtual scene, described herein. In thesame manner in which the processor system 505 generates one or morepackets having the audio data 505, the processor system 505 or thecommunication device 1126 (FIG. 11B) generates one or more packets byapplying a protocol, such as the network communication protocol, thewired protocol, or the wireless protocol, to the image frame data andsends the one or more packets to the display device 1250. For example,with reference to FIG. 11A, the processor system 505 applies the networkcommunication protocol to the image frame data to generate one or morenetwork packets and sends the one or more network packets via thecomputer network 1102 to the router and modem 1104 (FIG. 11A). Therouter and modem 1104 processes the one or more network packets havingthe image frame data in the same manner in which the router and modem1104 processes the one or more network packets having the audio data 805to obtain the image frame data from the one or more network packets,applies the wireless protocol to the image frame data to generate one ormore wireless packets, and sends the one or more wireless packets to thecommunication device 1252. As another example, with reference to FIG.11B, the communication device 1126 of the game console 1122 applies thewireless protocol to the image frame data to generate one or morewireless packets, and sends the wireless packets to the communicationdevice 1252 of the display device 1250. As another example, withreference to FIG. 11C, the cellular network 1144 receives one or morenetwork packets having the image frame data via the computer network1102 from the processor system 505 and applies the network communicationprotocol to extract the image frame data from the one or more networkpackets, and applies the cellular communication protocol to the imageframe data to generate one or more packets. The cellular network 1144sends the one or more packets having the image frame data to the smartphone 1142.

Referring back to FIG. 12B, the communication device 1152 receives theone or more packets having the image frame data and applies a protocol,such as the cellular communication protocol, the wired protocol, or thewireless protocol, to extract or obtain the image frame data from theone or more packets, and sends the image frame data to the displayscreen 1112. Upon receiving the image frame data, the display screen1112 displays the coaching scene or the virtual scene, described herein.

FIG. 13 is a flow diagram conceptually illustrating various operationswhich are performed for streaming a cloud video game to a client device,in accordance with implementations of the disclosure. Examples of theclient device include a game controller, a smart phone, a game console,and a computer. A game server 1302 executes a video game and generatesraw (uncompressed) video 1304 and audio 1306. The game event data 510 orthe game event data 504 or a combination thereof is an example of arecording of combination of the video 1004 and audio 1306. The gameserver 1302 is an example of the processor system 505. The video 1304and audio 1306 are captured and encoded for streaming purposes, asindicated at reference 1308 in the illustrated diagram. The encodingprovides for compression of the video and audio streams to reducebandwidth usage and optimize the gaming experience. Examples of encodingformats include H.265/MPEG-H, H.264/MPEG-4, H.263/MPEG-4, H.262/MPEG-2,WMV, VP6/7/8/9, etc.

Encoded audio 1310 and encoded video 1312 are further packetized intonetwork packets, as indicated at reference numeral 1314, for purposes oftransmission over a computer network 1320, which is an example of thecomputer network 1102 (FIG. 11A). In some embodiments, the networkpacket encoding process also employs a data encryption process, therebyproviding enhanced data security. In the illustrated implementation,audio packets 1316 and video packets 1318 are generated for transportover the computer network 1320.

The game server 1302 additionally generates haptic feedback data 1322,which is also packetized into network packets for network transmission.In the illustrated implementation, haptic feedback packets 1324 aregenerated for transport over the computer network 1320.

The foregoing operations of generating the raw video and audio and thehaptic feedback data are performed on the game server 1302 of a datacenter, and the operations of encoding the video and audio, andpacketizing the encoded audio/video and haptic feedback data fortransport are performed by the streaming engine of the data center. Asindicated at reference 1320, the audio, video, and haptic feedbackpackets are transported over the computer network. As indicated atreference 1326, the audio packets 1316, video packets 1318, and hapticfeedback packets 1324, are disintegrated, e.g., parsed, etc., by theclient device 602 (FIG. 6) to extract encoded audio 1328, encoded video1330, and haptic feedback data 1322 at the client device 602 from thenetwork packets. If data has been encrypted, then the data is alsodecrypted. The encoded audio 1328 and encoded video 1330 are thendecoded by the client device, as indicated at reference 1334, togenerate client-side raw audio and video data for rendering on a displaydevice 1340 of the client device 602. The haptic feedback data 1322 isprocessed by the processor of the client device 602 to produce a hapticfeedback effect at a controller device 1342 or other interface device,e.g., the HMD, etc., through which haptic effects can be rendered. Thecontroller device 1342 is an example of the game controller 1. Oneexample of a haptic effect is a vibration or rumble of the controllerdevice 1342.

It will be appreciated that a video game is responsive to user inputs,and thus, a similar procedural flow to that described above fortransmission and processing of user input, but in the reverse directionfrom client device to server, is performed. As shown, the controllerdevice 1342 or another input device, e.g., the body part of the user 1,etc., or a combination thereof generates input data 1348. Any of thecontrol input signals 616, 623, 630, 638, and 642 (FIG. 6) is an exampleof the input data 1348. The input data 1348 is packetized at the clientdevice for transport over the computer network 1320 to the data center.Input data packets 1346 are unpacked and reassembled by the game server1302 to define the input data 1348 on the data center side. The inputdata 1348 is fed to the game server 1302, which processes the input data1348 to generate a game state of the game.

During transport via the computer network 1320 of the audio packets1316, the video packets 1318, and haptic feedback packets 1324, in someembodiments, the transmission of data over the computer network 1320 ismonitored to ensure a quality of service. For example, networkconditions of the computer network 1320 are monitored as indicated byreference 1350, including both upstream and downstream networkbandwidth, and the game streaming is adjusted in response to changes inavailable bandwidth. That is, the encoding and decoding of networkpackets is controlled based on present network conditions, as indicatedby reference 1352.

FIG. 14 is a block diagram of an embodiment of a game console 1400 thatis compatible for interfacing with a display device of the client deviceand is capable of communicating via the computer network 1320 with agame hosting system, such as the processor system 505 (FIG. 5). The gameconsole 1122 (FIG. 11) is an example of the game console 1400. The gameconsole 1400 is located within a data center A or is located at alocation at which the user 1 is located. In some embodiments, the gameconsole 1400 is used to execute a game that is displayed on an HMD 1405.The game console 1400 is provided with various peripheral devicesconnectable to the game console 1400. The game console 1400 has a cellprocessor 1428, a dynamic random access memory (XDRAM) unit 1426, aReality Synthesizer graphics processor unit 1430 with a dedicated videorandom access memory (VRAM) unit 1432, and an input/output (I/O) bridge1434. The game console 1400 also has a Blu Ray® Disk read-only memory(BD-ROM) optical disk reader 1440 for reading from a disk 1440 a and aremovable slot-in hard disk drive (HDD) 1436, accessible through the I/Obridge 1434. Optionally, the game console 1400 also includes a memorycard reader 1438 for reading compact flash memory cards, memory Stick®memory cards and the like, which is similarly accessible through the I/Obridge 1434. The I/O bridge 1434 also connects to USB 2.0 ports 1424, agigabit Ethernet port 1422, an IEEE 802.11b/g wireless network (Wi-Fi™)port 1420, and a Bluetooth® wireless link port 1418 capable ofsupporting Bluetooth connections.

In operation, the I/O bridge 1434 handles all wireless, USB and Ethernetdata, including data from a game controller and from the HMD 1405. Forexample, when the user 1 is playing the game generated by execution of aportion of a game code, such as the game program 528 (FIG. 11A), the I/Obridge 1434 receives input data or an input signal, described herein,from the game controllers 1342 or 1403 and/or from the HMD 1405 via aBluetooth link and directs the input data to the cell processor 1428,which updates a current state of the game accordingly. As an example, acamera within the HMD 1405 captures a gesture of the user 1 to generatean image representing the gesture. The game controller 1342 is anexample of the game controller 1, which is an example of the HHC.

The wireless, USB and Ethernet ports also provide connectivity for otherperipheral devices in addition to the game controllers 1342 and 1403 andthe HMD 1405, such as, for example, a remote control 1404, a keyboard1406, a mouse 1408, a portable entertainment device 1410, such as, e.g.,a Sony Playstation Portable® entertainment device, etc., a video camera,such as, e.g., an EyeToy® video camera 1412, etc., a microphone headset1414, and a microphone 1415. The portable entertainment device 1410 isan example of a game controller. In some embodiments, such peripheraldevices are connected to the game console 1400 wirelessly, for example,the portable entertainment device 1410 communicates via a Wi-Fi™ ad-hocconnection, whilst the microphone headset 1414 communicates via aBluetooth link. The microphone headset 1414 is an example of the headphone 803 (FIG. 8C).

The provision of these interfaces means that the game console 1400 isalso potentially compatible with other peripheral devices such asdigital video recorders (DVRs), set-top boxes, digital cameras, portablemedia players, Voice over Internet protocol (IP) telephones, mobiletelephones, printers and scanners.

In addition, a legacy memory card reader 1416 is connected to the gameconsole 1400 via the USB port 1424, enabling the reading of memory cards1448 of a kind used by the game console 1400. The game controllers 1342and 1403 and the HMD 1405 are operable to communicate wirelessly withthe game console 1400 via the Bluetooth link 1418, or to be connected tothe USB port 1424, thereby also receiving power by which to chargebatteries of the game controller 1342 and 1403 and the HMD 1405. In someembodiments, each of the game controllers 1342 and 1403 and the HMD 1405includes a memory, a processor, a memory card reader, permanent memory,such as, e.g., flash memory, etc., light emitters such as, e.g., anilluminated spherical section, light emitting diodes (LEDs), or infraredlights, etc., microphone and speaker for ultrasound communications, anacoustic chamber, a digital camera, an internal clock, a recognizableshape, such as, e.g., a spherical section facing the game console 1400,and wireless devices using protocols, such as, e.g., Bluetooth, Wi-Fi,etc.

The game controller 1342 is a controller designed to be used with twohands by the user 1, and game controller 1403 is a single-handcontroller with an attachment. The HMD 1405 is designed to fit on top ofa head and/or in front of eyes of the user 1. In addition to one or moreanalog joysticks and conventional control buttons, each game controller1342 and 1403 is susceptible to three-dimensional locationdetermination. Similarly, the HMD 1405 is susceptible tothree-dimensional location determination. Consequently, in someembodiments, gestures and movements by the user 1 of the game controller1342 and 1403 and of the HMD 1405 are translated as inputs to a game inaddition to or instead of conventional button or joystick commands.Optionally, other wirelessly enabled peripheral devices, such as, e.g.,the Playstation™ Portable device, etc., are used as a controller. In thecase of the Playstation™ Portable device, additional game or controlinformation, e.g., control instructions or number of lives, etc., isprovided on a display screen of the device. In some embodiments, otheralternative or supplementary control devices are used, such as, e.g., adance mat (not shown), a light gun (not shown), a steering wheel andpedals (not shown), bespoke controllers, etc. Examples of bespokecontrollers include a single or several large buttons for arapid-response quiz game (also not shown).

The remote control 1404 is also operable to communicate wirelessly withthe game console 1400 via the Bluetooth link 1418. The remote control1404 includes controls suitable for the operation of the Blu Ray DiskBD-ROM reader 1440 and for navigation of disk content.

The Blu Ray™ Disk BD-ROM reader 1440 is operable to read CD-ROMscompatible with the game console 1400, in addition to conventionalpre-recorded and recordable CDs, and so-called Super Audio CDs. The BluRay™ Disk BD-ROM reader 1440 is also operable to read digital videodisk-ROMs (DVD-ROMs) compatible with the game console 1100, in additionto conventional pre-recorded and recordable DVDs. The Blu Ray™ DiskBD-ROM reader 1440 is further operable to read BD-ROMs compatible withthe game console 1400, as well as conventional pre-recorded andrecordable Blu-Ray Disks.

The game console 1400 is operable to supply audio and video, eithergenerated or decoded via the Reality Synthesizer graphics unit 1430,through audio connectors 1450 and video connectors 1452 to a display andsound output device 1442, such as, e.g., a monitor or television set,etc., having a display screen 1444 and one or more loudspeakers 1446, orto supply the audio and video via the Bluetooth® wireless link port 1118to the display device of the HMD 1405. The audio connectors 1450, invarious embodiments, include conventional analogue and digital outputswhilst the video connectors 1452 variously include component video,S-video, composite video, and one or more High Definition MultimediaInterface (HDMI) outputs. Consequently, video output may be in formatssuch as phase alternating line (PAL) or National Television SystemCommittee (NTSC), or in 2220p, 1080i or 1080p high definition. Audioprocessing, e.g., generation, decoding, etc., is performed by the cellprocessor 1408. An operating system of the game console 1400 supportsDolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and thedecoding of 7.1 surround sound from Blu-Ray® disks. The display andsound output device 1442 is an example of the display device 1 (FIG. 1).

In some embodiments, a video camera, e.g., the video camera 1412, etc.,comprises a single charge coupled device (CCD), an LED indicator, andhardware-based real-time data compression and encoding apparatus so thatcompressed video data is transmitted in an appropriate format such as anintra-image based motion picture expert group (MPEG) standard fordecoding by the game console 1400. An LED indicator of the video camera1412 is arranged to illuminate in response to appropriate control datafrom the game console 1400, for example, to signify adverse lightingconditions, etc. Some embodiments of the video camera 1112 variouslyconnect to the game console 1400 via a USB, Bluetooth or Wi-Ficommunication port. Various embodiments of a video camera include one ormore associated microphones and also are capable of transmitting audiodata. In several embodiments of a video camera, the CCD has a resolutionsuitable for high-definition video capture. In use, images captured bythe video camera are incorporated within a game or interpreted as gamecontrol inputs. In another embodiment, a video camera is an infraredcamera suitable for detecting infrared light.

In various embodiments, for successful data communication to occur witha peripheral device, such as, for example, a video camera or remotecontrol via one of the communication ports of the game console 1400, anappropriate piece of software, such as, a device driver, etc., isprovided.

In some embodiments, the aforementioned system devices, including thegame console 1400, the HHC, and the HMD 1405 enable the HMD 1405 todisplay and capture video of an interactive session of a game. Thesystem devices initiate an interactive session of a game, theinteractive session defining interactivity between the user 1 and otherusers and the game. The system devices further determine an initialposition and orientation of the HHC, and/or the HMD 1405 operated by theuser 1. The game console 1400 determines a current state of a game basedon the interactivity between the user 1 and the game. The system devicestrack a position and orientation of the HHC and/or the HMD 1405 duringan interactive session of the user 1 with a game. The system devicesgenerate a spectator video stream of the interactive session based on acurrent state of a game and the tracked position and orientation of theHHC and/or the HMD 1405. In some embodiments, the HHC renders thespectator video stream on a display screen of the HHC. In variousembodiments, the HMD 1405 renders the spectator video stream on adisplay screen of the HMD 1405.

With reference to FIG. 15, a diagram illustrating components of an HMD1502 is shown. The HMD 1502 is an example of the HMD 1405 (FIG. 14). TheHMD 1502 includes a processor 1500 for executing program instructions. Amemory device 1502 is provided for storage purposes. Examples of thememory device 1502 include a volatile memory, a non-volatile memory, ora combination thereof. A display device 1504 is included which providesa visual interface, e.g., display of image frames generated from savedata, etc., that any the user 1 (FIG. 1) views. A battery 1506 isprovided as a power source for the HMD 1502. A motion detection module1508 includes any of various kinds of motion sensitive hardware, such asa magnetometer 1510, an accelerometer 1512, and a gyroscope 1514.

An accelerometer is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis models are availableto detect magnitude and direction of the acceleration in differentdirections. The accelerometer is used to sense inclination, vibration,and shock. In one embodiment, three accelerometers 1512 are used toprovide the direction of gravity, which gives an absolute reference fortwo angles, e.g., world-space pitch and world-space roll, etc.

A magnetometer measures a strength and a direction of a magnetic fieldin a vicinity of the HMD 1502. In some embodiments, three magnetometers1510 are used within the HMD 1502, ensuring an absolute reference forthe world-space yaw angle. In various embodiments, the magnetometer isdesigned to span the earth magnetic field, which is ±80 microtesla.Magnetometers are affected by metal, and provide a yaw measurement thatis monotonic with actual yaw. In some embodiments, a magnetic field iswarped due to metal in the real-world environment, which causes a warpin the yaw measurement. In various embodiments, this warp is calibratedusing information from other sensors, e.g., the gyroscope 1514, a camera1516, etc. In one embodiment, the accelerometer 1512 is used togetherwith magnetometer 1510 to obtain the inclination and azimuth of the HMD1502.

A gyroscope is a device for measuring or maintaining orientation, basedon the principles of angular momentum. In one embodiment, instead of thegyroscope 1514, three gyroscopes provide information about movementacross the respective axis (x, y and z) based on inertial sensing. Thegyroscopes help in detecting fast rotations. However, the gyroscopes, insome embodiments, drift overtime without the existence of an absolutereference. This triggers resetting the gyroscopes periodically, whichcan be done using other available information, such aspositional/orientation determination based on visual tracking of anobject, accelerometer, magnetometer, etc.

The camera 1516 is provided for capturing images and image streams of areal-world environment, e.g., room, cabin, natural environment, etc.,surrounding any of the users 1-3. In various embodiments, more than onecamera is included in the HMD 1502, including a camera that isrear-facing, e.g., directed away from the user 1 when the user 1 isviewing the display of the HMD 1502, etc., and a camera that isfront-facing, e.g., directed towards the user 1 when the user 1 isviewing the display of the HMD 1502, etc. Additionally, in severalembodiments, a depth camera 1518 is included in the HMD 1502 for sensingdepth information of objects in the real-world environment.

The HMD 1502 includes speakers 1520 for providing audio output. Also, amicrophone 1522 is included, in some embodiments, for capturing audiofrom the real-world environment, including sounds from an ambientenvironment, and speech made by the user 1, etc. The HMD 1502 includes atactile feedback module 1524, e.g., a vibration device, etc., forproviding tactile feedback to the user 1. In one embodiment, the tactilefeedback module 1524 is capable of causing movement and/or vibration ofthe HMD 1502 to provide tactile feedback to the user 1.

LEDs 1526 are provided as visual indicators of statuses of the HMD 1502.For example, an LED may indicate battery level, power on, etc. A cardreader 1528 is provided to enable the HMD 1502 to read and writeinformation to and from a memory card. A USB interface 1530 is includedas one example of an interface for enabling connection of peripheraldevices, or connection to other devices, such as other portable devices,computers, etc. In various embodiments of the HMD 1502, any of variouskinds of interfaces may be included to enable greater connectivity ofthe HMD 1502.

A Wi-Fi™ module 1532 is included for enabling connection to the Internetvia wireless networking technologies. Also, the HMD 1502 includes aBluetooth™ module 1534 for enabling wireless connection to otherdevices. A communications link 1536 is also included, in someembodiments, for connection to other devices. In one embodiment, thecommunications link 1536 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1536utilizes any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1538 are included to provide an input interfacefor the user 1 (FIG. 1). Any of various kinds of input interfaces areincluded, such as buttons, touchpad, joystick, trackball, etc. Anultra-sonic communication module 1540 is included, in variousembodiments, in the HMD 1502 for facilitating communication with otherdevices via ultra-sonic technologies.

Bio-sensors 1542 are included to enable detection of physiological datafrom the user 1. In one embodiment, the bio-sensors 1542 include one ormore dry electrodes for detecting bio-electric signals of the user 1through the user 1's skin.

The foregoing components of HMD 1502 have been described as merelyexemplary components that may be included in HMD 1502. In variousembodiments, the HMD 1502 include or do not include some of the variousaforementioned components.

FIG. 16 illustrates an embodiment of an Information Service Provider(INSP) architecture. INSPs 1302 delivers a multitude of informationservices to the user 1 geographically dispersed and connected via acomputer network 1606, e.g., a LAN, a WAN, or a combination thereof,etc. The computer network 1102 (FIG. 11A) is an example of the computernetwork 1606. An example of the WAN includes the Internet and an exampleof the LAN includes an Intranet. The user 1 operates a client device1620-1, another user 2 operates another client device 1620-2, and yetanother user 3 operates yet another client device 1620-3. The clientdevice 1620-1 is an example of the client device 602 (FIG. 6).

In some embodiments, each client device 1620-1, 1620-2, and 1620-3includes a central processing unit (CPU), a display, and an input/output(I/O) interface. Examples of each client device 1620-1, 1620-2, and1620-3 include a personal computer (PC), a mobile phone, a netbook, atablet, a gaming system, a personal digital assistant (PDA), the gameconsole 1400 and a display device, the HMD 1502 (FIG. 15), the gameconsole 1400 and the HMD 1502, a desktop computer, a laptop computer,and a smart television, etc. In some embodiments, the INSP 1602recognizes a type of a client device and adjusts a communication methodemployed.

In some embodiments, an INSP delivers one type of service, such as stockprice updates, or a variety of services such as broadcast media, news,sports, gaming, etc. Additionally, the services offered by each INSP aredynamic, that is, services can be added or taken away at any point intime. Thus, an INSP providing a particular type of service to aparticular individual can change over time. For example, the clientdevice 1620-1 is served by an INSP in near proximity to the clientdevice 1620-1 while the client device 1620-1 is in a home town of theuser 1, and client device 1620-1 is served by a different INSP when theuser 1 travels to a different city. The home-town INSP will transferrequested information and data to the new INSP, such that theinformation “follows” the client device 1620-1 to the new city makingthe data closer to the client device 1620-1 and easier to access. Invarious embodiments, a master-server relationship is established betweena master INSP, which manages the information for the client device1620-1, and a server INSP that interfaces directly with the clientdevice 1620-1 under control from the master INSP. In some embodiments,data is transferred from one ISP to another ISP as the client device1620-1 moves around the world to make the INSP in better position toservice client device 1620-1 be the one that delivers these services.

The INSP 1602 includes an Application Service Provider (ASP) 1608, whichprovides computer-based services to customers over the computer network1606. Software offered using an ASP model is also sometimes calledon-demand software or software as a service (SaaS). A simple form ofproviding access to a computer-based service, e.g., customerrelationship management, etc., is by using a standard protocol, e.g., ahypertext transfer protocol (HTTP), etc. The application softwareresides on a vendor's server and is accessed by each client device1620-1, 1620-2, and 1620-3 through a web browser using a hypertextmarkup language (HTML), etc., by a special purpose client softwareprovided by the vendor, and/or other remote interface, e.g., a thinclient, etc.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe computer network 1606. The users 1-3 do not need to be an expert inthe technology infrastructure in the “cloud” that supports them. Cloudcomputing is divided, in some embodiments, in different services, suchas Infrastructure as a Service (IaaS), Platform as a Service (PaaS), andSoftware as a Service (SaaS). Cloud computing services often providecommon business applications online that are accessed from a webbrowser, while the software and data are stored on the servers. The termcloud is used as a metaphor for the computer network 1606, e.g., usingservers, storage and logic, etc., based on how the computer network 1606is depicted in computer network diagrams and is an abstraction for thecomplex infrastructure it conceals.

Further, the INSP 1602 includes a game processing provider (GPP) 1610,also sometime referred to herein as a game processing server, which isused by the client devices 1620-1, 1620-2, and 1620-3 to play single andmultiplayer video games. Most video games played over the computernetwork 1606 operate via a connection to a game server. Typically, gamesuse a dedicated server application that collects data from the clientdevices 1620-1, 1620-2, and 1620-3 and distributes it to other clientsthat are operated by other users. This is more efficient and effectivethan a peer-to-peer arrangement, but a separate server is used to hostthe server application. In some embodiments, the GPP 1610 establishescommunication between the client devices 1620-1, 1620-2, and 1620-3,which exchange information without further relying on the centralizedGPP 1610.

Dedicated GPPs are servers which run independently of a client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are a method of hosting game servers for most PC-basedmultiplayer games. Massively multiplayer online games run on dedicatedservers usually hosted by the software company that owns the game title,allowing them to control and update content.

A broadcast processing server (BPS) 1612, sometimes referred to hereinas a broadcast processing provider, distributes audio or video signalsto an audience. Broadcasting to a very narrow range of audience issometimes called narrowcasting. A final leg of broadcast distribution ishow a signal gets to the client devices 1620-1, 1620-2, and 1620-3, andthe signal, in some embodiments, is distributed over the air as with aradio station or a television station to an antenna and receiver, orthrough a cable television or cable radio or “wireless cable” via thestation. The computer network 1606 also brings, in various embodiments,either radio or television signals to the client devices 1620-1, 1620-2,and 1620-3, especially with multicasting allowing the signals andbandwidth to be shared. Historically, broadcasts are delimited, inseveral embodiments, by a geographic region, e.g., national broadcasts,regional broadcasts, etc. However, with the proliferation of high-speedInternet, broadcasts are not defined by geographies as content can reachalmost any country in the world.

A storage service provider (SSP) 1614 provides computer storage spaceand related management services. The SSP 1614 also offers periodicbackup and archiving. By offering storage as a service, the clientdevices 1620-1, 1620-2, and 1620-3 use more storage compared to whenstorage is not used as a service. Another major advantage is that theSSP 1614 includes backup services and the client devices 1620-1, 1620-2,and 1620-3 will not lose data if their hard drives fail. Further, aplurality of SSPs, in some embodiments, have total or partial copies ofthe data received from the client devices 1620-1, 1620-2, and 1620-3,allowing the client devices 1620-1, 1620-2, and 1620-3 to access data inan efficient way independently of where the client devices 1620-1,1620-2, and 1620-3 are located or of types of the clients. For example,the user 1 accesses personal files via a home computer, as well as via amobile phone while the user 1 is on the move.

A communications provider 1616 provides connectivity to the clientdevices 1620-1, 1620-2, and 1620-3. One kind of the communicationsprovider 1616 is an Internet service provider (I6P) which offers accessto the computer network 1606. The ISP connects the client devices1620-1, 1620-2, and 1620-3 using a data transmission technologyappropriate for delivering Internet Protocol datagrams, such as dial-up,digital subscriber line (DSL), cable modem, fiber, wireless or dedicatedhigh-speed interconnects. The communications provider 1616 alsoprovides, in some embodiments, messaging services, such as e-mail,instant messaging, and short message service (SMS) texting. Another typeof a communications Provider is a network service provider (NSP), whichsells bandwidth or network access by providing direct backbone access tothe computer network 1606. Examples of network service providers includetelecommunications companies, data carriers, wireless communicationsproviders, Internet service providers, cable television operatorsoffering high-speed Internet access, etc.

A data exchange 1618 interconnects the several modules inside INSP 1302and connects these modules to the client devices 1620-1, 1620-2, and1620-3 via computer network 1606. The data exchange 1618 covers, invarious embodiments, a small area where all the modules of INSP 1602 arein close proximity, or covers a large geographic area when the differentmodules are geographically dispersed. For example, the data exchange1602 includes a fast Gigabit Ethernet within a cabinet of a data center,or an intercontinental virtual LAN.

In some embodiments, communication between the server system and theclient devices 1620-1 through 1620-3 may be facilitated using wirelesstechnologies. Such technologies may include, for example, 5G wirelesscommunication technologies. 5G is the fifth generation of cellularnetwork technology. 5G networks are digital cellular networks, in whichthe service area covered by providers is divided into small geographicalareas called cells. Analog signals representing sounds and images aredigitized in the telephone, converted by an analog-to-digital converterand transmitted as a stream of bits. All the 5G wireless devices in acell communicate by radio waves with a local antenna array and low powerautomated transceiver (transmitter and receiver) in the cell, overfrequency channels assigned by the transceiver from a pool offrequencies that are reused in other cells. The local antennas areconnected with the telephone network and the Internet by a highbandwidth optical fiber or wireless backhaul connection. As in othercell networks, a mobile device crossing from one cell to another isautomatically transferred to the new cell. It should be understood that5G networks are just an example type of communication network, andembodiments of the disclosure may utilize earlier generation wireless orwired communication, as well as later generation wired or wirelesstechnologies that come after 5G.

It should be noted that in various embodiments, one or more features ofsome embodiments described herein are combined with one or more featuresof one or more of remaining embodiments described herein.

Embodiments described in the present disclosure may be practiced withvarious computer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. In oneimplementation, the embodiments described in the present disclosure arepracticed in distributed computing environments where tasks areperformed by remote processing devices that are linked through awire-based or wireless network.

With the above embodiments in mind, it should be understood that, in oneimplementation, the embodiments described in the present disclosureemploy various computer-implemented operations involving data stored incomputer systems. These operations are those requiring physicalmanipulation of physical quantities. Any of the operations describedherein that form part of the embodiments described in the presentdisclosure are useful machine operations. Some embodiments described inthe present disclosure also relate to a device or an apparatus forperforming these operations. The apparatus is specially constructed forthe required purpose, or the apparatus is a general-purpose computerselectively activated or configured by a computer program stored in thecomputer. In particular, in one embodiment, various general-purposemachines are used with computer programs written in accordance with theteachings herein, or it may be more convenient to construct a morespecialized apparatus to perform the required operations.

In an implementation, some embodiments described in the presentdisclosure are embodied as computer-readable code on a computer-readablemedium. The computer-readable medium is any data storage device thatstores data, which is thereafter read by a computer system. Examples ofthe computer-readable medium include a hard drive, a network-attachedstorage (NAS), a ROM, a RAM, a compact disc ROM (CD-ROM), aCD-recordable (CD-R), a CD-rewritable (CD-RW), a magnetic tape, anoptical data storage device, a non-optical data storage device, etc. Asan example, a computer-readable medium includes computer-readabletangible medium distributed over a network-coupled computer system sothat the computer-readable code is stored and executed in a distributedfashion.

Moreover, although some of the above-described embodiments are describedwith respect to a gaming environment, in some embodiments, instead of agame, other environments, e.g., a video conferencing environment, etc.,is used.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing embodiments described in the present disclosurehave been described in some detail for purposes of clarity ofunderstanding, it will be apparent that certain changes andmodifications can be practiced within the scope of the appended claims.Accordingly, the present embodiments are to be considered asillustrative and not restrictive, and the embodiments are not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

1. A method for processing a self-coaching interface, comprising,identifying a gameplay event during gameplay by a user, wherein thegameplay event is tagged as falling below a skill threshold while a gameis being played by the user; generating a recording for a window of timefor the gameplay event; processing game telemetry for the recording ofthe gameplay event, wherein the game telemetry is used to identify aprogression of interactive actions before the gameplay event for thewindow of time; and generating overlay content in the self-coachinginterface, wherein the overlay content is applied to one or more imageframes of the recording when viewed via the self-coaching interface, theoverlay content appearing in the one or more image frames during aplayback of the recording, wherein the overlay content provides hintsfor increasing a skill of the user to be above the skill threshold forthe tagged gameplay event.
 2. The method of claim 1, wherein the windowof time includes a time period before the gameplay event, a time of thegameplay event, and a time period after the gameplay event.
 3. Themethod of claim 1, wherein the window of time changes with the gameplayevent and other gameplay events occurring before the gameplay event. 4.The method of claim 1, wherein the game telemetry is processed toidentify a reason for the skill of the user to fall below the threshold.5. The method of claim 4, wherein the game telemetry includes one ormore game states that are processed to identify a virtual object that isnot in the recording for the time window.
 6. The method of claim 1,wherein the progression of interactive actions includes a movement of avirtual object that is not in the recording for the time window.
 7. Themethod of claim 6, wherein the overlay content includes the virtualobject.
 8. The method of claim 6, wherein the overlay content includes aframe that is placed around the virtual object.
 9. The method of claim1, further comprising: generating a timeline scrubber; receiving aselection of a segment of the timeline scrubber; and modifying theself-coaching interface and the overlay content based on a selection ofthe segment.
 10. The method of claim 1, further comprising generatinganother self-coaching interface for display with the self-coachinginterface, wherein the other self-coaching interface is associated withgame play of another user and is of the same type as the gameplay event.11. The method of claim 1, further comprising generating anotherself-coaching interface for display with the self-coaching interface,wherein the other self-coaching interface is associated with a differentgaming session of game play of the user and is of the same type as thegameplay event.
 12. The method of claim 1, further comprising trainingan artificial intelligence model to determine whether to initiate acoaching session in which the overlay content is overlaid on theself-coaching interface.
 13. The method of claim 1, wherein theself-coaching interface excludes one or more virtual objects that aredisplayed in a virtual scene during occurrence of the gameplay event.14. The method of claim 1, wherein the self-coaching interface includesone or more virtual objects that are not displayed in a virtual sceneduring occurrence of the game play event.
 15. A server system forprocessing a self-coaching interface, comprising, a processor configuredto: identify a gameplay event during gameplay by a user, wherein thegameplay event is tagged as falling below a skill threshold while a gameis being played by the user; generate a recording for a window of timefor the gameplay event; process game telemetry for the recording of thegameplay event, wherein the game telemetry is used to identify aprogression of interactive actions before the gameplay event for thewindow of time; and generate overlay content in the self-coachinginterface, wherein the overlay content is applied to one or more imageframes of the recording when viewed via the self-coaching interface, theoverlay content appearing in the one or more image frames during aplayback of the recording, wherein the overlay content provides hintsfor increasing a skill of the user to be above the skill threshold forthe tagged gameplay event; and a memory device coupled to the processorfor storing the recording of the game play event.
 16. The server systemof claim 15, wherein the processor is configured to generate anotherself-coaching interface for display with the self-coaching interface,wherein the other self-coaching interface is associated with game playof another user and is related to the gameplay event.
 17. The serversystem of claim 15, wherein the processor is configured to generateanother self-coaching interface for display with the self-coachinginterface, wherein the other self-coaching interface is associated witha different gaming session of game play of the user and with thegameplay event.
 18. The server system of claim 15, wherein the window oftime changes with the gameplay event and other gameplay events occurringbefore the gameplay event.
 19. A system for processing a self-coachinginterface, comprising, a client device configured to be used by a userto facilitate generation of gameplay event while playing a game; aserver coupled to the client device via a computer network, wherein theserver is configured to: identify the gameplay event, wherein thegameplay event is tagged as falling below a skill threshold while a gameis being played by the user; generate a recording for a window of timefor the gameplay event; process game telemetry for the recording of thegameplay event, wherein the game telemetry is used to identify aprogression of interactive actions before the gameplay event for thewindow of time; and generate overlay content in the self-coachinginterface, wherein the overlay content is applied to one or more imageframes of the recording when viewed via the self-coaching interface, theoverlay content appearing in the one or more image frames during aplayback of the recording, wherein the overlay content provides hintsfor increasing a skill of the user to be above the skill threshold forthe tagged gameplay event.
 20. The system of claim 19, wherein theserver system is configured to generate another self-coaching interfacefor display with the self-coaching interface, wherein the otherself-coaching interface is associated with game play of another user andis related to the gameplay event.